Modular switched fabric for data storage systems

ABSTRACT

Systems, methods, apparatuses, and software for data storage systems are provided herein. In one example, a data storage system is provided that includes a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured to store data and retrieve the data stored on associated storage media responsive to storage operations. The data storage system includes one or more processing modules comprising one or more processors communicatively coupled to the plurality of storage drives over a PCIe fabric comprised of one or more PCIe switches. The processors are configured to share a PCIe address space associated with the PCIe fabric for transfer of the storage operations to appropriate ones of the processors that manage ones of the plurality of storage drives.

RELATED APPLICATIONS

This application hereby claims the benefit of and priority to U.S.Provisional Patent Application 62/015,927, titled “INTER-MODULEREDUNDANCY FOR COMPUTE NODES,” filed Jun. 23, 2014; U.S. ProvisionalPatent Application 62/015,931, titled “INTRA-MODULE REDUNDANCY FORCOMPUTE NODES,” filed Jun. 23, 2014; U.S. Provisional Patent Application62/015,936, titled “IDENTIFICATION OF STORAGE SLEDS,” filed Jun. 23,2014; U.S. Provisional Patent Application 62/015,939, titled “FRONT ENDETHERNET LOAD BALANCING,” filed Jun. 23, 2014; U.S. Provisional PatentApplication 62/015,942, titled “SSD INITIALIZATION SPOOFING,” filed Jun.23, 2014; U.S. Provisional Patent Application 62/015,910, titled“CLUSTER-WIDE DEVICE VISIBILITY,” filed Jun. 23, 2014; U.S. ProvisionalPatent Application 62/015,947, titled “TCP FAILOVER MAINTAININGCONNECTION STATE,” filed Jun. 23, 2014; U.S. Provisional PatentApplication 62/015,967, titled “CLUSTERING OF STORAGE NODES OVER PCIE,”filed Jun. 23, 2014; U.S. Provisional Patent Application 62/015,977,titled “LOGICAL STORAGE UNIT PRESENTATION OF STORAGE NODE OVER ISCSI,”filed Jun. 23, 2014; U.S. Provisional Patent Application 62/015,990,titled “NVME TARGET ABSTRACTION/EMULATION OVER A VIRTUAL LUN,” filedJun. 23, 2014; and U.S. Provisional Patent Application 62/018,090,titled “VLUN ABSTRACTION OVER PCIE PHYSICAL INTERFACE AS NVME TARGET,”filed Jun. 27, 2014; which are hereby incorporated by reference in theirentirety.

BACKGROUND

Computer systems typically include bulk storage systems, such asmagnetic disk drives, optical storage devices, tape drives, or solidstate storage drives, among other storage systems. As storage needs haveincreased in these computer systems, networked storage systems have beenintroduced which store large amounts of data in a storage environmentphysically separate from end user computer devices. These networkedstorage systems typically provide access to bulk data storage over oneor more network interfaces to end users or other external systems. Inaddition to storage of data, remote computing systems include variousprocessing systems that can provide remote computing resources to endusers. These networked storage systems and remote computing systems canbe included in high-density installations, such as rack-mountedenvironments.

However, as the densities of networked storage systems and remotecomputing systems increase, various physical limitations can be reached.These limitations include density limitations based on the underlyingstorage technology, such as in the example of large arrays of rotatingmagnetic media storage systems. These limitations can also includecomputing density limitations based on the various physical spacerequirements for network interconnect as well as the large spacerequirements for environmental climate control systems.

In addition to physical space limitations, these bulk storage systemshave been traditionally limited in the number of devices that can beincluded per host, which can be problematic in storage environmentswhere higher capacity, redundancy, and reliability is desired. Theseshortcomings can be especially pronounced with the increasing datastorage and retrieval needs in networked, cloud, and enterpriseenvironments.

Overview

Systems, methods, apparatuses, and software for data storage systems areprovided herein. In one example, a data storage system is provided thatincludes a plurality of storage drives each comprising a PeripheralComponent Interconnect Express (PCIe) interface, and configured to storedata and retrieve the data stored on associated storage media responsiveto storage operations. The data storage system includes one or moreprocessing modules comprising one or more processors communicativelycoupled to the plurality of storage drives over a PCIe fabric comprisedof one or more PCIe switches. The processors are configured to share aPCIe address space associated with the PCIe fabric for transfer of thestorage operations to appropriate ones of the processors that manageones of the plurality of storage drives.

In another example, a method of operating a data storage system isprovided. The method includes, in a plurality of storage drives, storingdata and retrieving the data stored on associated storage mediaresponsive to storage operations received over a Peripheral ComponentInterconnect Express (PCIe) interface. The method includes, in one ormore processing modules comprising one or more processorscommunicatively coupled to the plurality of storage drives over a PCIefabric comprised of one or more PCIe switches, sharing a PCIe addressspace associated with the PCIe fabric for transfer of the storageoperations to appropriate ones of the processors that manage ones of theplurality of storage drives.

In another example, a modular data storage system is provided. Themodule data storage system includes a plurality of storage modules eachcomprising a plurality of solid state drives and configured tocommunicate over a switched Peripheral Component Interconnect Express(PCIe) fabric. The module data storage system includes a plurality ofprocessing modules each comprising one or more processors that areconfigured to manage storage operations for a designated subset of theplurality of storage modules over the switched PCIe fabric. The moduledata storage system includes a plurality of network interfacesindividually associated with ones of the processing modules andconfigured to communicate with the associated ones of the processingmodules over the switched PCIe fabric. The module data storage systemincludes one or more PCIe switches configured to establish the switchedPCIe fabric over a plurality of PCIe links to interconnect the storagemodules to the processing modules and interconnect the processingmodules to the network interfaces. The processors are configured toshare a PCIe address space associated with the switched PCIe fabric fortransfer of received storage operations to appropriate ones of theprocessors that manage ones of the plurality of storage drives.

BRIEF DESCRIPTION OF THE DRAWINGS

Many aspects of the disclosure can be better understood with referenceto the following drawings. The components in the drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating the principles of the present disclosure. Moreover, in thedrawings, like reference numerals designate corresponding partsthroughout the several views. While several embodiments are described inconnection with these drawings, the disclosure is not limited to theembodiments disclosed herein. On the contrary, the intent is to coverall alternatives, modifications, and equivalents.

FIG. 1 is a system diagram illustrating a storage system.

FIG. 2 is a block diagram illustrating a storage system.

FIG. 3 is a block diagram illustrating a storage module.

FIG. 4 is a block diagram illustrating control modules.

FIG. 5 is a block diagram illustrating a processing module.

FIG. 6 is a system diagram illustrating a storage system.

FIG. 7 is a system diagram illustrating a storage system.

FIG. 8 is a system diagram illustrating a storage system.

FIG. 9A is a system diagram illustrating a storage system.

FIG. 9B is a system diagram illustrating a storage system.

FIG. 10 is a diagram illustrating a memory allocation table.

FIG. 11 is a system diagram illustrating a storage system.

FIG. 12 is a system diagram illustrating a storage system.

FIG. 13 is a system diagram illustrating a storage system.

FIG. 14A is a system diagram illustrating a storage system.

FIG. 14B is a system diagram illustrating a storage system.

FIG. 15 is a flow diagram illustrating operation of a storage system.

FIG. 16 is a system diagram illustrating a storage system.

FIG. 17 is a system diagram illustrating a storage system.

FIG. 18 is a block diagram illustrating a control processor.

DETAILED DESCRIPTION

FIG. 1 is a system diagram illustrating storage system 100. Storagesystem 100 includes three different module types in FIG. 1, namelystorage sleds 110, interconnect modules 120, and processing modules 130.Although this example shows many storage sleds, 2 interconnect modules,and 6 processing modules. Any number of sleds or modules can beincludes, such as 48 storage sleds or 64 storage sleds, along with adifferent number of interconnect or processing modules. Some examplescan distribute functionality of each interconnect module 120 among twoor more modules. Additionally, power supply modules and associated powerand control distribution links can also be included, but are omitted inFIG. 1 for clarity.

A module typically comprises physical support structure and enclosurethat includes circuitry, printed circuit boards, semiconductor systems,and structural elements. The modules are insertable and removable from arackmount style of enclosure. In some examples, the elements of FIG. 1are included in a 3U chassis for mounting in a larger rackmountenvironment. It should be understood that the elements of FIG. 1 can beincluded in any physical mounting environment, and need not include anyassociated enclosures or rackmount elements.

Holdup circuitry can be included on each sled 110 to provide power tothe associated sled when input power has been lost or removed for thesled. In some examples, the sled is removed from an associated matingconnector and input power is lost due to the removal. In other examples,power is lost to system 100, such as during a facility power outage orwhen an associated power supply fails. Similar holdup circuitry can beincluded on the other various modules of system 100. Specifically,holdup circuitry can be included on interconnect modules 120 andprocessing modules 130.

A plurality of storage sleds 110 are included in system 100. Eachstorage sled 110 includes one or more storage drives, such as four eachshown in FIG. 3. Each storage sled 110 also includes PeripheralComponent Interconnect Express (PCIe) switches, processors, and controlsystem elements. PCIe switches of each storage sled 110 communicate withone or more on-sled storage drives over associated PCIe links. PCIeswitches of each storage sled 110 also are communicatively coupled to anon-sled processor or control system for traffic statistics retrieval andstatus monitoring, among other operations. PCIe switches of each storagesled 110 communicate over one or more PCIe links 140 with an associatedPCIe switch 121 of an interconnect module 120.

Each PCIe switch 121 of interconnect modules 120 communicate overassociated PCIe links 142 with associated PCIe switch 132 of one or moreprocessing modules 130. PCIe switch 132 communicates with one or moreassociated processing systems 131 over PCIe links 136 as well as overone or more cross-connect PCIe links 143. Interconnect modules 120 alsoeach include a plurality of PCIe switches 122 for interconnectingprocessor modules, such as processor modules 130. PCIe switches 122 areincluded for processor module cross-connect, and communicate with onesof PCIe switches 133 in associated processing modules 130 over processormodule cross-connect links 141. PCIe switches 133 communicate with onesof processing systems 131 over PCIe links 134.

Each processing module 130 communicates over one or more PCIe links 135through PCIe switches 133 with external expansion cards or external PCIeports. In some examples, the external expansion cards include networkinterface cards for communicating over TCP/IP (Transmission ControlProtocol (TCP)/Internet Protocol) networks or for carrying iSCSI(Internet Small Computer System Interface), Non-Volatile Memory HostController Interface Specification (NVMHCI) traffic, or NVMe (NVMExpress) traffic, among other network traffic types. These packet linksare illustrated by packet network links 144, which include one or morelinks 144A-144 n. External access to storage system 100 is provided overones of packet network links 144, such as for end user access to datastored on storage sleds 110.

Each processing module 130 can also communicate with other processingmodules, such as those in other storage assemblies or 3U enclosures,over one or more inter-module packet network interfaces 145, whichinclude one or more links 145A-145 n. In some examples, inter-modulepacket network interfaces 145 include network interface cards forcommunicating over Ethernet or TCP/IP (Transmission Control Protocol(TCP)/Internet Protocol) networks for exchanging storage packets betweenprocessing modules. Further operation of inter-module storage packetexchange over Ethernet is discussed in the examples herein.

In some examples, external expansion cards comprising external PCIeconnectors include one or more external PCIe interfaces for carryingNVMe traffic. These PCIe links are illustrated by links 146. Externalaccess to storage system 100 is provided over ones of PCIe links 146 and147, such as for end user access to data stored on storage sleds 110.Storage system 100 is accessible by an external system, such as a serveror computer system, over one or more PCIe interfaces. These PCIeinterfaces are represented in FIG. 1 by PCIe link 146 and PCIe link 147.Further external PCIe links can be included, and two are shown in FIG. 1for clarity. PCIe link 147 includes a PCIe link provided by PCIe switch122 on I/O module 120. PCIe link 146 includes a PCIe link provided by anexpansion card that interfaces with processing module 130 over PCIeswitch 133. PCIe links 146 and 147 can include external PCIe connectorsand cabling, such as employed in PCIe direct connect.

The PCIe switches discussed herein can comprise PCIe crosspointswitches, which logically interconnect various ones of the associatedPCIe links based at least on the traffic carried by each PCIe link. Insome examples, logical segregation of the PCIe fabric is achieved usingnon-transparent ports of the PCIe switches. Selected PCIe switch portscan be configured as non-transparent (NT) or transparent ports. An NTport can allow some logical isolation between endpoints, much like abridge, while a transparent port does not allow logical isolation, andhas the effect of connecting endpoints in a purely switchedconfiguration. Access over an NT port or ports can include additionalhandshaking between the PCIe switch and the initiating endpoint toselect a particular NT port or to allow visibility through the NT port.

In other examples, logical segregation of the PCIe fabric is achievedusing domain-based routing of the PCIe switches. A domain-based PCIesignaling distribution can be included in the PCIe switches which allowssegregation of PCIe ports of a PCIe switch according to user-definedgroups. These logical segregation or groups can be used to form thefront-end and back-end data planes discussed herein, or can be used forestablishing routing and redundancy among the various PCIe switches,forming a mesh fabric of PCIe traffic in system 100.

In FIG. 1, PCIe switches 121 and 132 (and associated PCIe links) areincluded in a back end data plane of system 100, and used for carryingstorage operations and storage data between storage sleds 110 andprocessing modules 130. PCIe switches 122 and 133 (and associated PCIelinks) are included in a front end data plane of system 100, and usedfor carrying storage operations and storage data, as well as usercontrol data and control signaling between processing modules. In someexamples, one or more Ethernet interfaces and associated links comprisea control plane of system 100, such as over interfaces 145 as well asother Ethernet interfaces, such as shown for Ethernet links 447-448 ofFIG. 4 below. The control plane can be used for control signalingbetween processing modules, management messaging, initializationmessaging, statistical data monitoring and gathering from controlprocessors, such as processor 450 of FIG. 4, among other control dataand information.

In an alternative arrangement of system 100, which can be applied to theexamples in further figures herein, a unified PCIe fabric is employedwhich does not physically distinguish among front-end data plane andback-end data planes. Instead, a common set of PCIe switches areemployed which can route PCIe traffic among various destinations basedon factors such as PCIe addressing, logic domains that are established,sideband PCIe messaging, or other techniques. In logical domainexamples, a PCIe switch can be dynamically programmed by any processingsystem 131 or control processor. This programming can be done over adiscrete control interface or link, or via specialized PCIe commands orsideband PCIe messaging, to segregate the PCIe links of the PCIe switchinto various logical groups. However, in many of the examples herein,separate front end and back end data planes formed by separate PCIefabrics are employed. It should be understood that either configurationis possible, such as unified data planes and unified PCIe fabrics, andthe examples herein do not limit the configuration to one particularconfiguration of PCIe fabric.

PCIe can support multiple bus widths, such as ×1, ×4, ×8, ×16, and ×32,with each multiple of bus width comprising an additional “lane” for datatransfer. PCIe also supports transfer of sideband signaling, such asSystem Management Bus (SMBus) interfaces and Joint Test Action Group(JTAG) interfaces, as well as associated clocks, power, andbootstrapping, among other signaling. Although PCIe is used in FIG. 1,it should be understood that different communication links or busses caninstead be employed, such as Ethernet, Serial Attached SCSI (SAS), NVMe,FibreChannel, Thunderbolt, Serial Attached ATA Express (SATA Express),among other interconnect, network, and link interfaces. Any of the linksin FIG. 1 can each use various communication media, such as air, space,metal, optical fiber, or some other signal propagation path, includingcombinations thereof. Any of the links in FIG. 1 can include any numberof PCIe links or lane configurations. Any of the links in FIG. 1 caneach be a direct link or might include various equipment, intermediatecomponents, systems, and networks. Any of the links in FIG. 1 can eachbe a common link, shared link, aggregated link, or may be comprised ofdiscrete, separate links.

In FIG. 1, any processing system 131 on any processing module 130 haslogical visibility to all storage drives in all storage sleds 110. Anyprocessing system 131 can transfer data for storage on any storage driveand retrieve data already stored on any storage drive. Thus, ‘m’ numberof storage drives can be coupled with ‘n’ number of processors to allowfor a large, scalable architecture with a high-level of redundancy anddensity.

To provide visibility of each processing system 131 to any storage sled110, various techniques can be employed. In a first example, aparticular processing system 131 manages (instantiates/binds) a subsetnumber of the total quantity of storage sleds, such as 16 storage drivesspanning 4 storage sleds, and handles transactions for that subset ofstorage drives, such as read and write transactions. Each processingsystem 131, however, has memory-mapped visibility to the storage drivesmanaged by any other processing system 131. When a transaction isdesired for a storage drive not managed by a particular processingsystem, the particular processing system uses the memory mapped accessto all storage drives for that transaction. The transaction can betransferred and transitioned to the appropriate processing system thatmanages that storage drive associated with the data of the transaction.The front end data plane, namely PCIe switches 122 and 133 are used totransfer data between processing systems so that a particular processingsystem or processor can store the data in the storage sled or sleds thatis managed by that particular processing system, even though the datamight be received over a network interface associated with a differentprocessing system.

In an example, the PCIe interfaces associated with each processingsystem 131 have 64-bit address spaces, which allows an addressable spaceof 2⁶⁴ bytes, leading to at least 16 exbibytes of byte-addressablememory. The 64-bit PCIe address space can shared by all processingsystems 131 for memory mapping to storage drives on storage sleds. Thus,while each particular processing system 131 actually manages a subset ofthe total storage drives on storage sleds, all processors 131 havevisibility to, and can initiate read/write transactions to, any ofstorage drives on storage sleds. A managing processing system 131 thatmanages a particular storage drives on storage sleds receives write/readtransactions and any associated data from an initiating processingsystem 131 by at least using the memory mapped PCIe address space.

In operation, such as a write operation, data can be received over anynetwork interface 144 by any processing system 131 of any processingmodule 130. For example, the write operation can be a write operationreceived over network link 144 from an end user employing an iSCSIprotocol or NVMe protocol. The processing system that receives the writeoperation determines if it physically manages the storage drive ordrives associated with the write operation, and if it does, then theprocessing system transfers the data for storage on the associatedstorage drives over back end data plane PCIe links 136. If theprocessing system determines that it does not physically manage thestorage drive or drives associated with the write operation, then theprocessing system transfers the write operation to another processingsled that includes the processing system that does manages the storagedrive or drives over cross connect links 134. Data striping can beemployed by any processing system to stripe data for a particular writetransaction over any number of storage drives, such as over all of thestorage sleds that include storage drives managed by the particularprocessing system.

NVMe (NVM Express) is an interface standard for mass storage devices,such as hard disk drives and solid state memory devices. NVMe cansupplant serial ATA (SATA) interfaces for interfacing with mass storagedevices in personal computers and server environments. However, theseNVMe interfaces are limited to one-to-one host-drive relationship,similar to SATA devices. In the examples discussed herein, a PCIeinterface is employed to transport NVMe traffic and present amulti-processor, multi-drive system as one or more NVMe virtual logicalunit numbers (VLUNs) over a PCIe interface.

In NVMe operations, such as an NVMe write operation, data can bereceived over any of PCIe links 141-143 and 146-147 by any processingsystem 131 of any processing module 130. For example, the writeoperation can be an NVMe write operation received over PCIe link 143from a device employing an NVMe protocol transported over a PCIeinterface. In another example, the write operation can be an NVMe writeoperation received over PCIe link 146 or 147 from an external deviceemploying an NVMe protocol transported over a PCIe interface. Theprocessing system that receives the write operation determines if itmanages the storage drive or drives associated with the write operation,and if it does, then the processing system transfers the data forstorage on the associated storage drives over PCIe links 142. If theprocessing system determines that it does not physically manage thestorage drive or drives associated with the write operation, then theprocessing system transfers the write operation to another processingsled that includes the processing system that does manages the storagedrive or drives over cross connect links 134. Data striping can beemployed by any processing system to stripe data for a particular writetransaction over any number of storage drives, such as over all of thestorage sleds that include storage drives managed by the particularprocessing system.

Any of processing systems 131 can present the storage resources ofstorage system 100 as a VLUN, such as NVMe VLUNs. In the example of FIG.1, a first VLUN is presented as VLUN 101 and a second VLUN is presentedas VLUN 102. Although these VLUNs are shown associated with differentprocessing systems on a single processing module 130, in other examplesa VLUN is presented by any processing system on any processing moduleand can storage traffic for the VLUN can be handled by any processingsystem on any processing module. The processing modules of system 100can present any number of VLUNs to an external system over a PCIeinterface, such as any of PCIe links 146-147. These VLUNs can bepresented as an NVMe target, such as NVMe target A or NVMe target B inFIG. 1. An NVMe target can present the storage resources of storagesystem 100 as a single storage target, such as emulating a singlestorage drive, over a PCIe interface. In this manner, a plurality ofstorage drives that comprise any number of storage sleds 110 can bepresented as a single NVMe target to an external system over a PCIeinterface. Any of the processing systems of system 100 can receive NVMestorage traffic, such as NVMe frames, and distribute these storagetransactions for handling by an assigned processing system within system100.

Some network storage protocols, such as iSCSI, allow for grouping ofdestination addresses into a logical unit. For example, iSCSI allows forportal groups which can associate more than one IP address with a singletarget. The target is a particular processor or particular networkinterface, even if that interface or processor has more than one networkaddress associated therewith. However, FIG. 1 as well as FIGS. 8 and 12illustrate forming a logical grouping of destination addresses for anetwork storage protocol using a plurality of processors or processingnodes that share a common storage space spread over a plurality ofstorage devices. Many processors can be included in a portal group, witheach processor configured to receive network storage traffic for any ofthe other processors and route the traffic accordingly to the correctprocessor, even though a single iSCSI portal group is presented to aniSCSI initiator. In specific examples, a PCIe fabric is employed betweenmany processors, network interfaces, and storage drives. These conceptscan apply to network storage protocols other than iSCSI.

In FIG. 1, more than one network interface is employed. Specifically,network interfaces 144 or 145 can be coupled to the plurality ofprocessing modules 130 shown in FIG. 1. Each network interface can haveone or more IP addresses associated therewith, and can be reachable overa packet network using the associated IP addresses. The various networkinterfaces of system 100 can be formed into an iSCSI portal group, whichallows an iSCSI initiator to transfer network storage transactions, suchas iSCSI transactions, over a packet network to any of networkinterfaces 144 or 145. As a specific example, FIG. 1 shows an iSCSIportal group being formed from external network links 144A-144 n. Thisportal group can be presented to network devices over network links144A-144 n, and the portal group can span the resources of system 100,such as one or more processing systems 131, one or more processingmodule 130, and one or more of storage sleds 110, among other elements.Further discussions on iSCSI portal groups are discussed in FIGS. 8 and12.

To show an example physical configuration of a storage system, such assystem 100, FIG. 2 is presented. FIG. 2 is a block diagram illustratingstorage system 200. The elements of FIG. 2 are shown as enclosed byenclosure 201 and physically mated to midplane 240. Midplane 240includes a chassis and a plurality of physical connectors for matingwith any associated storage sleds 210, interconnect modules 220,processing modules 230, or external interfaces 235. Midplane 240comprises one or more printed circuit boards, connectors, physicalsupport members, chassis elements, structural elements, and associatedlinks as metallic traces or optical links for interconnecting thevarious elements of FIG. 2. Midplane 240 can function as a backplane,but instead of having sleds or modules mate on only one side as inbackplane examples, midplane 240 has sleds or modules that mate on atleast two sides. Elements of FIG. 2 can correspond to similar elementsof FIG. 1, such as storage sled 110, interconnect module 120, processingmodule 130, and the expansion/external connectors 144-146, althoughvariations are possible.

FIG. 2 shows all elements included in a 3U enclosure 201. The enclosurecan instead be of any multiple of a standardized computer rack height,such as 1U, 2U, 3U, 4U, and the like, and can include associatedchassis, physical supports, cooling systems, mounting features, cases,and other enclosure elements. Typically, each sled or module will fitinto associated groove features included in a chassis portion ofenclosure 201 to slide into a predetermined slot and guide an edgeconnector associated with each sled to mate with an associated socketconnector on midplane 240. Storage sleds 210 each have an associatedconnector 260. Interconnect modules 220 each have an associatedconnector 261. Interconnect modules 220 also each have one or morecluster interconnect links 267, which in this example are PCIe links.Cluster interconnect links 267 are employed to interconnect 3Uenclosures between each other using PCIe links. Control modules 225 eachhave an associated connector 269. In some examples, ones of controlmodules 225 and interconnect modules 220 are included in the samemodule. Processing modules 230 each have an associated connector 262.Processing modules 230 each have one or more associated external links266 for communicating with external systems, such as management systems,end user devices, or other computing systems, including other enclosuressimilar to enclosure 201. External links 266 can comprise Ethernet,SFP+, or other links and connectors. External interfaces module 235 eachhave an associated connector 268. External interfaces 235 provideexternal access to the storage contents of enclosure 201, such as forend user devices or external systems. Network links 265 can be providedby external interfaces 235, which can comprises Ethernet, TCP/IP,Infiniband, iSCSI, NVMe, or other external interfaces. In operation,external interfaces 235 each is communicatively coupled with anassociated processing module, as pictured in FIG. 2. Enclosure 201enables hot-swapping of any of the sleds and can include other featuressuch as power lights, activity indicators, external administrationinterfaces, and the like.

In some examples, enclosure 201 includes a chassis and midplane that canaccommodate a flexible configuration and arrangement of sleds andassociated circuit cards. Although FIG. 2 illustrates storage sledsmating on one side of midplane 240 and various modules mating on anotherside of midplane 240, it should be understood that other configurationsare possible. Enclosure 201 can include a chassis to accommodate any ofthe following configurations, either in front-loaded or rear-loadedconfigurations:

-   -   one or more sleds that contain four M.2 SSDs each    -   one or more sleds containing 2×HHHL cards (half-height        half-length PCIe cards) that can comprise PCIe storage cards,        PCIe network adaptors, or host bus adaptors    -   one or more sleds with 2×FHHL cards (full-height half-length        PCIe cards) that can comprise graphics cards or graphics        processing units (GPUs)    -   one or more sleds with 1× PCIe cards (full-height full-length        PCIe cards) that comprise processing modules, which can comprise        NVIDIA Tesla or Intel Phi processor cards    -   one or more sleds containing 4×2.5-inch PCIe SSDs    -   interconnect modules, interposer modules, and control modules

Additionally, power and associated power control signaling for thevarious sleds of enclosure 201 is provided by one or more power supplynodes 250 over associated links 263. Although power supply nodes 250 areshown as included in enclosure 201 in FIG. 2, it should be understoodthat power supply nodes 250 can instead be included in separateenclosures, such as separate 1U enclosures. Each power supply node 250also includes power link 264 for receiving power from power sources,such as AC or DC input power. Additionally, power holdup circuitry canbe included in holdup modules 251 which can deliver holdup power overlinks 274 responsive to power loss over link 264 or from a failure ofpower supply nodes 350.

Power holdup circuitry can be included on each sled or module of FIGS. 1and 2. This power holdup circuitry can be used to provide interim powerto the associated sled or module during power interruptions, such aswhen main input or system power is lost from a power source.Additionally, during use of holdup power, processing portions of eachsled or module can be employed to selectively power down portions ofeach sled according to usage statistics, among other considerations.This holdup circuitry can provide enough power to commit in-flight writedata during power interruptions or power loss events. These powerinterruption and power loss events can include loss of power from apower source, or can include removal of a sled or module from anassociated socket or connector on midplane 240. The holdup circuitry caninclude capacitor arrays, super-capacitors, ultra-capacitors, batteries,fuel cells, flywheels, or other energy storage components, along withany associated power control, conversion, regulation, and monitoringcircuitry.

Storage sleds 210 or any of the insertable modules in FIG. 2 can eachprovide self-power during power interruption events. Also, storage sleds210 can each redistribute power to other storage sleds or modules. Thisredistributed power can be transferred to other storage sleds 210 or toother modules in FIG. 2, such as interconnect module 220, control module225, processing module 230, or external interfaces 235. Once anyin-flight write data has been committed to non-volatile memory of astorage sled, then excess or remaining holdup power can be redistributedto other modules. In a first example power is lost to 3U enclosure 201,such as loss of input power over links 264 or failure of power supplies250, among other failures or loss. In this example, storage sleds 210can self-power for a period of time or redistribute power to othermodules or sleds. In an example operation, one storage sled 210transfers holdup power to another of storage sleds 210, such as overpower links of the associated modules. In another example operation,storage sled 210 transfers power to a processing module or interconnectmodule, among others. This redistribution of power from storage sleds orother modules for use by other storage sleds or other modules can occurover a shared power link or power bus included in midplane 240.Directional control of power flow can be provided by circuitry in eachmodule or sled which allows the associated module to receive input poweror to redistribute power over the same power link.

In another example operation 271, a storage sled is removed frommidplane 240 and thus has any input power lost due to physical removal.The removed storage sled can detect removal, such as by using pull-up orpull-down resistors, detection pins, link capacitance or impedancedetection, or detection of loss of input power, among other detectionmethods. If in-flight write data has not yet been committed during lossof power, then associated holdup circuitry can power the removed storagesled. In sled removal examples, redistribution of power to other sledsor modules is not likely due to removal and thus remaining holdup powernot redistributed to other sleds or modules. Instead, excess orremaining holdup power can be bled off to an associated power sink. Thispower sink can include a bleed resistor or resistor array which convertsthe excess holdup power to heat. In other examples, one or moreindicator lights can be included in bleed-off circuitry and theindicator lights can remain lit during a bleed off procedure to indicatepower remains on the sled as well as to bleed power by powering theindicator lights. Once the sled is reinserted into midplane 240, inputpower is restored and the sled can receive power from midplane 240 aswell as charge any holdup circuitry for the next power loss event.

FIG. 3 illustrates a block diagram of storage module 310, as an exampleof any of storage sleds 110 of FIG. 1 or storage sleds 210 of FIG. 2.FIG. 3 also illustrates an example physical configuration of storagemodule 310 as shown for circuit card assembly 350. Storage module 310includes four storage drives 311, PCIe switch 312, processor 320, powercontrol module 321, and holdup circuit 322. Power control module 321distributes power to each element of storage module 310 over associatedpower links 330-332. Power control module 321 can selectivelyenable/disable power for each power link. Further communication linkscan be included for intra-sled communication between the variouselements of storage module 310.

PCIe switch 312 communicates with an interconnect module (not pictured)over links 340. Links 340 are included in a back end data plane fortransferring user data, such as that discussed for FIG. 1. Each of links340 comprises a PCIe link with four lanes, namely a “×4” PCIe link. Morethan one PCIe link 340 is provided for load balancing, redundancy, andfailover protection for storage module 310. In some examples, PCIeswitch 312 has links 340 connected to non-transparent (NT) interfaces orports, where one or more host systems (such as a processor on aprocessing module) can interact with storage drives 311 in a redundantor failover configuration. PCIe switch 312 also communicates with fourstorage drives 111 over associated ×4 PCIe links 341.

Processor 320 communicates over at least sideband links 349. Sidebandlinks 349 can include Universal Serial Bus (USB), SMBus, JTAG,Inter-Integrated Circuit (I2C), controller area network bus (CAN), orany other communication interface, and in some examples is provided overportions of PCIe links 340. In this example, processor 320 includes I2Cinterface 325 and USB interface 326 for communication over sidebandlinks 349. I2C interface 325 and USB interface 326 can be included inseparate circuitry or included in similar elements as processor 320.Processor 320 and PCIe switch 312 can communicate over an associatedcommunication link 333, which can be an I2C or a PCIe link, among otherlink types.

Each storage drive 311 comprises a solid state drive (SSD) in thisexample, and communicates with external systems over an associated PCIeinterface included in each storage drive 311. The solid state storagemedia of storage drives 311 can comprise flash memory, static RAM, NANDflash memory, NOR flash memory, memristors, or other solid state media.Instead of or in addition to solid state media, each storage drive 311can comprise magnetic storage, such as hard disk drives, tape drives,magnetoresistive memory devices, and the like, or can comprise opticalstorage, such as phase change memory. Each storage drive 311 can receiveread transactions and write transactions issued by a host system, suchas a processor of a processing sled node. Responsive to a readtransaction, storage drive 311 can retrieve data identified by the readtransaction and transfer the data for delivery to the associated host.Responsive to a write transaction, storage drive 311 can write data thataccompanies the write transaction to storage media associated withstorage drive 311.

In some examples, each storage drive 311 comprises a circuit cardassembly (CCA) which is separate from CCA 350 and with a mini-PCIExpress connector or other connector that interfaces with a connector onCCA 350. CCA 350 comprises one or more printed circuit boards 351 thatcouple to the various elements of storage module 310. In other examples,each storage drive 311 comprises one or more flash memory chips with aPCIe interface which is soldered onto CCA 350. In yet other examples,each storage drive 311 comprises one or more separate solid state diskdrives or magnetic hard disk drives along with associated enclosures andcircuitry. PCIe switch 312 comprises a PCIe cross connect switch forestablishing switched connections between any PCIe interfaces handled byPCIe switch 312. In some examples, PCIe switch 312 comprises a PLXTechnology PEX8725 10-port, 24 lane PCIe switch chip.

Processor 320 comprises one or more microprocessors, processing devices,multi-core processors, processing circuitry, or other processing system.Processor 320 can include one or more non-transitory memory devices,such as RAM, solid state storage, or other memory to store instructionsthat are executable by processor 320 to operate as discussed herein. Insome examples, processor 320 comprises an ARM microcontroller, ARMmicroprocessor, field-programmable gate array (FPGA), applicationspecific integrated circuit (ASIC), application specific processor, orother microprocessor or processing elements. Processor 320 can compriseany processing elements discussed below for processing system 510 ofFIG. 5. Processor 320 can monitor usage statistics, traffic status, orother usage information through link 333. PCIe switch 312 can track thisusage information during normal operation and data transfer with storagedrives 311, and processor 320 can retrieve this usage information asneeded over link 333.

Power control module 321 includes circuitry to selectively provide powerto any of the elements of storage module 310. Power control module 321can receive control instructions from processor 320 or over any of PCIelinks 340. In some examples, power control module 321 comprisesprocessing elements discussed above for processor 320, or is included inthe elements of processor 320. Power control module 321 can receivepower over power link 323 as a power source for the various elements ofstorage module 310. Holdup circuit 322 includes energy storage devicesfor storing power received over power link 323 for use during powerinterruption events, such as loss of source power. Holdup circuit 322can include capacitance storage devices, such as an array of capacitors.Power can be accepted by module 310 when input power is available, suchas from a mating connector or midplane. Power can be redistributed toother modules by module 310 over link 323 when input power is notavailable, such as during power interruption events. When module 310 isremoved from a mating connector, then power can be bled off intoassociated power sink circuitry. Although one power link 323 is shown,it should be understood that more than one link can be included, such asseparate input and output links or separate links for different voltagelevels.

FIG. 4 is a block diagram illustrating input/output (I/O) module 420 asexamples of interconnect module 120 of FIG. 1 or interconnection module220 and control module 235 of FIG. 2. It should be understood that theelements of module 420 can be combined onto a single module, such asshown in FIG. 1, or included in separate modules.

I/O module 420 includes at least two back end data plane PCIe switches421, at least two front end data plane PCIe switches 424, power controlmodule 422, and holdup circuit 423. Power control module 422 distributespower to each element of I/O module 420 over associated power links431-433. Power control module 422 can selectively enable/disable powerfor each power link. Further communication links can be included forintra-sled communication between the various elements of I/O module 420.

PCIe switches 421, PCIe links 440, and PCIe links 442 are included in aback end data plane, or back end, of a storage system, such as thatillustrated in FIG. 1. Each PCIe switch 421 communicates with two ormore storage sleds (not pictured) over PCIe links 440, and with two ormore processing modules (not pictured) over PCIe links 442. Each oflinks 440 comprises a PCIe link with four lanes, namely a “×4” PCIelink. Each of links 442 comprises a PCIe link with eight lanes, namely a“×8” PCIe link. In some examples, each PCIe switch 421 communicates withsix processing modules, such as shown in FIG. 1. Each PCIe link can alsoinclude sideband signaling, such as SMBus, JTAG, I2C, CAN, or any othercommunication interface, and in some examples is provided over separatelinks. Each PCIe switch 421 comprises a PCIe cross connect switch forestablishing switched connections between any PCIe interfaces handled byeach PCIe switch 421. In some examples, each PCIe switch 421 comprises aPLX Technology PEX8796 24-port, 96 lane PCIe switch chip.

PCIe switches 424 and PCIe links 443 are included in a front end dataplane, or front end, of a storage system, such as that illustrated inFIG. 1. Each PCIe switch 424 communicates with many processing modulesover PCIe links 443, and with each other over PCIe links 432. Each oflinks 443 and 432 comprises a PCIe link with eight lanes, namely a “×8”PCIe link. In some examples, each PCIe switch 424 communicates with sixprocessing modules, such as shown in FIG. 1. Each PCIe link can alsoinclude sideband signaling, such as SMBus, JTAG, I2C, CAN, or any othercommunication interface, and in some examples is provided over separatelinks. Each PCIe switch 424 comprises a PCIe cross connect switch forestablishing switched connections between any PCIe interfaces handled byeach PCIe switch 424. In some examples, each PCIe switch 424 comprises aPLX Technology PEX8796 24-port, 96 lane PCIe switch chip.

Power control module 422 includes circuitry to selectively provide powerto any of the elements of I/O module 420. Power control module 422 canreceive control instructions from a processing module over any of PCIelinks 442. In some examples, power control module 422 comprisesprocessing elements discussed above for processor 320 of FIG. 3. Powercontrol module 422 can receive power over power link 445 as a powersource for the various elements of I/O module 420. Holdup circuit 423includes energy storage devices for storing power received over powerlink 445 for use during power interruption events, such as loss ofsource power. Holdup circuit 423 can include capacitance storagedevices, such as an array of capacitors. Further discussion of examplesof power control circuitry is found below.

Power can be accepted by module 420 when input power is available, suchas from a mating connector or midplane. Power can be redistributed toother modules by module 420 over link 445 when input power is notavailable, such as during power interruption events. When module 420 isremoved from a mating connector, then power can be bled off intoassociated power sink circuitry. Although one power link 445 is shown,it should be understood that more than one link can be included, such asseparate input and output links or separate links for different voltagelevels. Also, although only module 420 shows power control circuitry andholdup circuitry in FIG. 4, similar circuitry and power links can beincluded in other control or interposer modules when module 420 iscomprised of separate modules.

I/O module 420 includes processor 450, network switch 451, USB hub 452,I2C interface 453, and universal asynchronous receiver/transmitter(UART) interface 454. Network switch 451 can include one or moreEthernet switches, including transceivers, transformers, isolationcircuitry, buffers, and the like. USB hub 452 includes USB hub circuitryfor fanning out a single host USB connection to many device USB links,and can include transceivers, processors, transformers, isolationcircuitry, buffers, and the like.

Processor 450 includes one or more microprocessors or microcontrollersalong with any associated storage memory. Processor 450 communicateswith processing modules over Ethernet control plane links 447 and 448.Processor 450 communicates with data storage sleds over sideband links441, 444, and 449 which can comprise USB links, I2C links, or seriallinks, among others. An interworking or interposing module canfacilitate communication by processor 450 over any of the sidebandlinks, such as though FPGA 455. FPGA 455 provides protocol translationbetween an interface of processor 405, such as Serial PeripheralInterfaces (SPI), and the various serial sideband interfaces. Forexample, FPGA 455 can convert communications received over an SPIinterface from processor 450 to communications of I2C interface 453 orcommunications of UART interface 454. Likewise, FPGA 455 can convertcommunications received over the sideband interfaces and transfer overan interface to processor 450. FPGA 455 can handle fan-out andreplication of various interface for communications to interface withmore than one storage sled over the associated sideband interfaces.

Processor 450 initializes data storage sleds, such as discussed abovefor FIG. 1. Processor 450 accumulates statistical data and usageinformation for each storage sled in a storage system. Processingmodules can retrieve this statistical data or usage information overEthernet link 447 via network switch 451 from processing modules or oversideband links from data storage sleds. Ethernet link 447 comprises acontrol plane, which can be used for transferring control informationand status information between processing modules. I/O module 420 alsoprovides expansion of the control plane to other 3U enclosures forcluster control plane interconnect over Ethernet link 448. A furtherexample of processor 450 can be seen in control processor 1800 in FIG.18.

PCIe switches 424 each include cluster interconnect interfaces 446 whichare employed to interconnect further I/O modules of other storagesystems in further enclosures, such as in other 3U enclosures seen inFIG. 2. Interfaces 446 comprise PCIe links with 16 lanes, namely “×16”PCIe links. Cluster interconnect provides PCIe interconnect betweenexternal systems, such as other storage systems, over associatedexternal connectors and external cabling. These connections can be PCIelinks provided by any of the include PCIe switches, among other PCIeswitches not shown, for interconnecting other I/O modules of otherstorage systems via PCIe links. The PCIe links used for clusterinterconnect can terminate at external connectors. These connections caninstead be PCIe links provided by any of the included PCIe switches,among other PCIe switches not shown, for interconnecting otherinterconnect modules of other storage systems via PCIe links. The PCIelinks used for cluster interconnect can terminate at externalconnectors, such as mini-Serial Attached SCSI (SAS) connectors which areemployed to carry PCIe signaling over mini-SAS cabling.

The PCIe links used for cluster interconnect can terminate at externalconnectors, such as mini-Serial Attached SCSI (SAS) connectors which areemployed to carry PCIe signaling over mini-SAS cabling. In furtherexamples, MiniSAS HD cables are employed that drive 12 Gb/s versus 6Gb/s of standard SAS cables. 12 Gb/s can support PCIe Gen 3. To furtherillustrate cluster interconnection, FIG. 16 is presented below.

FIG. 5 is a block diagram illustrating processing module 530, as anexample of processing modules 130 of FIG. 1. Processing module 530includes two or more processing systems 510, at least one PCIe switch532, PCIe switches 540, network interfaces 534, power control module535, and holdup circuit 536. Power control module 535 distributes powerto each element of processing module 530 over associated power links537-539. Power control module 535 can selectively enable/disable powerfor each power link. Further communication links can be included forintra-sled communication between the various elements of processingmodule 530.

Each processing system 510 further includes processor 511 and storagesystem 512. In some examples, network interfaces 534-535 are included inprocessing system 510, but network interfaces 534-535, or portionsthereof, can be provided by separate circuitry and elements, such asseparate PCIe expansion cards. Each processing system 510 can send andreceive storage operations, such as storage packets over any of networkinterfaces 534-535, such as from external systems, end user devices, orfrom other processing systems of a cluster.

Each processor can communicate over an associated Ethernet sidebandsignaling link 556, such as with various microprocessors/controllers orpower control nodes on other sleds or with interconnect, interposer, orcontrol modules to retrieve statistical data or usage information. Links556 can comprise Ethernet interfaces, or can comprise SMBus, JTAG, I2C,CAN, or any other communication interfaces, and in some examples isprovided over separate links. Links 556 can be provided using externalnetwork interfaces, such as network interface cards or adapterscommunicatively coupled over ones of PCIe links 543. Each processor 511also includes at least one PCIe interface, such as a PCIe transceiverand communication circuitry for communicating over associated PCIe links533 and 541. The PCIe interface of each processor 511 can include alarge number of PCIe lanes which are subdivided between narrower PCIelinks, such as a ×16 interface that is subdivided among two ×8 links. Insome examples, the PCIe interfaces are integrated into a single-chip dieof processor 511. In other examples, the PCIe interface is provided overa separate microchip transceiver which can communicate with anassociated processor 511 over another communication interface, such as afront-side bus of processor 511 or peripheral hub chip interface.

Processor 511 can comprise one or more microprocessors and otherprocessing circuitry that retrieves and executes software 513 fromstorage system 512. Processor 511 can be implemented within a singleprocessing device but can also be distributed across multiple processingdevices or sub-systems that cooperate in executing program instructions.Examples of processor 511 include general purpose central processingunits, application specific processors, and logic devices, as well asany other type of processing device, combinations, or variationsthereof. In some examples, processor 511 comprises an Intel or AMDmicroprocessor, ARM microprocessor, FPGA, ASIC, application specificprocessor, or other microprocessor or processing elements.

Storage system 512 can comprise any non-transitory computer readablestorage media capable of storing software 513 that is executable byprocessor 511. Storage system 512 can include volatile and nonvolatile,removable and non-removable media implemented in any method ortechnology for storage of information, such as computer readableinstructions, data structures, program modules, or other data. Storagesystem 512 can be implemented as a single storage device but can also beimplemented across multiple storage devices or sub-systems co-located ordistributed relative to each other. Storage system 512 can compriseadditional elements, such as a controller, capable of communicating withprocessor 511. Examples of storage media include random access memory,read only memory, magnetic disks, optical disks, flash memory, virtualmemory and non-virtual memory, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store the desired information and that canbe accessed by an instruction execution system, as well as anycombination or variation thereof.

Software 513 can be implemented in program instructions and among otherfunctions can, when executed by processing system 510 in general orprocessor 511 in particular, direct processor 511 to operate ascommanded by software 513. Software 513 can include processes, programs,or components, such as operating system software, database software, orapplication software. Software 513 can also comprise firmware or someother form of machine-readable processing instructions executable byelements of processing system 501, such as processor 511. Encodingsoftware 513 on storage system 512 can transform the physical structureof storage system 512. The specific transformation of the physicalstructure can depend on various factors in different implementations ofthis description. Examples of such factors can include, but are notlimited to the technology used to implement the storage media of storagesystem 512 and whether the computer-storage media are characterized asprimary or secondary storage. For example, if the computer-storage mediaare implemented as semiconductor-based memory, software 513 cantransform the physical state of the semiconductor memory when theprogram is encoded therein. For example, software 513 can transform thestate of transistors, capacitors, or other discrete circuit elementsconstituting the semiconductor memory. A similar transformation canoccur with respect to magnetic or optical media. Other transformationsof physical media are possible without departing from the scope of thepresent description, with the foregoing examples provided only tofacilitate this discussion.

Processing module 530 also interfaces with one or more networkinterfaces 534-535 to communicate over one or more associated networklinks 554-555. One or more network interfaces 534-535 arecommunicatively coupled to an associated processing system 510 orprocessor 511 via associated PCIe links 543 in FIG. 5. External accessto processing module 530, and the storage sleds managed thereby, can beprovided over ones of packet network links 554. Communication betweenprocessing modules or processors can be provided over ones of packetnetwork links 555. In this example, packet network links 554-555 eachcomprises an Ethernet link for carrying various packet communications,such as Transmission Control Protocol/Internet Protocol (TCP/IP)communications, NVMe communications, or iSCSI communications, althoughother packet types can be employed. The network interfaces can compriseEthernet interfaces, IP interfaces, T1 interfaces, or other local orwide area network communication interfaces which can communicate over acommunication link. Examples of communication transceivers includenetwork interface card equipment, receivers, transmitters, modems, andother communication circuitry. In some examples, network links 554comprise a different bandwidth or transfer speed than network links 555.

PCIe switch 532 handles back end data plane or “back side” traffic forprocessing module 530 for storage and retrieval of data. PCIe switch 532communicates with storage sleds through one or more interconnect modules(not pictured) over PCIe links 552, and with ones of processors 531 overPCIe links 533. Each of links 552 and 533 comprises a PCIe link witheight lanes, namely a “×8” PCIe link. PCIe switch 532 also includesprocessing module cross connect links 533, which in this example arefour ×8 PCIe links. This processing module cross connect 533 allows aPCIe 532 switch on another processing module to handle back end dataplane traffic for any processing module for load balancing andbottleneck prevention. PCIe switch 532 also can include PCIe links 552for further processing module cross connect. PCIe switch 532 comprises aPCIe cross connect switch for establishing switched connections betweenany PCIe interface handled by PCIe switch 532. In some examples, PCIeswitch 532 comprises a PLX Technology PEX8796 24-port, 96 lane PCIeswitch chip.

PCIe switches 540 handle front end data plane or “front side” trafficfor processing module 530 for communications between processors ofdifferent processing modules. PCIe switches 540 communicate with otherprocessing modules through one or more interconnect modules (notpictured) over PCIe links 542, and with ones of processors 531 over PCIelinks 541. Each of links 542 and 541 comprises a PCIe link with eightlanes, namely a “×8” PCIe link. PCIe switches 540 also includes externalmodule links 543, which in this example are four ×8 PCIe links. PCIeswitches 540 comprise a PCIe cross connect switch for establishingswitched connections between any PCIe interface handled by PCIe switches540. In some examples, PCIe switches 540 each comprise a PLX TechnologyPEX8796 24-port, 96 lane PCIe switch chip.

Power control module 535 includes circuitry to selectively provide powerto any of the elements of processing module 530. Power control module535 can receive control instructions from a processor 511 overassociated links 537-539 or additional communication links. In someexamples, power control module 535 comprises processing elementsdiscussed above for processor 320 of FIG. 3. Power control module 535can receive power over power link 557 as a power source for the variouselements of processing module 530. Holdup circuit 536 includes energystorage devices for storing power received over power link 557 for useduring power interruption events, such as loss of source power. Holdupcircuit 536 can include capacitance storage devices, such as an array ofcapacitors.

FIG. 6 is a system diagram illustrating storage system 600, which can bean example of some elements of system 100 in FIG. 1. Storage system 600includes storage sleds 610 and 615, I/O modules 620 and 625, processingmodule 630, and interface modules 640-641. Storage sled 610 includes SSDdrives 611 and PCIe switch 612. Storage sled 615 includes SSD drives 616and PCIe switch 617. Although two storage sleds are included in FIG. 6,it should be understood that any number of storage sleds can beincluded. Also, although four SSD drives are included in each storagesled, it should be understood that any number of SSD drives can beincluded. I/O module 620 includes PCIe switch 621, and I/O module 625includes PCIe switch 626. Although one PCIe switch is shown for each I/Omodule in FIG. 6, it should be understood that any number can beincluded. Processing module 630 includes processors 631-632 and PCIeswitches 633-634, and further elements can be included as discussedabove.

Interface module 640 communicates over various interfaces, namely anetwork interface 642 for TCP/IP, iSCSI, or NVMe traffic, and aprocessing module PCIe interface 656. Interface module 641 communicatesover at least three interfaces, namely a network interface 645 forTCP/IP, iSCSI, or NVMe traffic, and a processing module PCIe interface657. Processing module 630 communicates with storage sleds 610 and 615over PCIe links 654 and 655 which are switched through associated PCIeswitches on associated I/O modules. Storage sled 610 communicates withdifferent I/O modules over respective PCIe links 650-651. Storage sled615 communicates with different I/O modules over respective PCIe links652-653. The elements of FIG. 6 form a storage system with multipleredundant PCIe links between processing modules and storage sleds by wayof I/O modules, such as those discussed herein.

In operation, a specific processor of a processing module will beconfigured to manage or “own” specific SSD drives on various storagesleds. For example, processor 631 can be configured to manage SSD drives611 of storage sled 610 while processor 632 can be configured to manageSSD drives 616 of storage sled 615. Other combinations can be used,including a subset of SSD drives of a particular storage sled managed bya processor and the remaining SSD drives of that particular storage sledmanaged by another processor. Managing an SSD drive comprises aprocessor initializing the SSD drive and handling read, write, or otherfile or object transactions for the SSD drive. Each processor canreference a data structure stored on a non-transitory computer readablemedium to determine which SSD drives are to be managed.

However, once a particular processor has been configured to manage aparticular SSD drive or set of SSD drives, that processor mightexperience a failure or interruption in normal operation. Responsive tothe failure of a processor, such as processor 631, another processor 632can be configured to take over management of the SSD drives previouslymanaged by the failed processor 631. To take over management of the SSDdrives previously managed by the failed processor 631, processor 632 caninitialize the SSD drives previously managed by the failed processor 631and begin managing file operations, such as read and write operations,for those SSD drives. Processor 632 can include failover table 680 whichindicates SSD drives for which processor 632 manages in the event of afailure of another processor. Failover table 680 can also indicate aprocessor or processors that processor 632 monitors to determine if anyfailures have occurred.

In FIG. 6, different pathways for data and other file transactions areshown. In a first pathway, namely normal pathway 690, interface module640 receives data packets corresponding to a file transaction (such asdata store or retrieve transactions) and transfers the data packets overPCIe switch 634 for delivery to processor 631. Processor 631 determinesthat it “owns” SSD drives that correspond to the file transaction andproceeds to handle the file transaction for the particular SSD drives,such as SSD drives 611. If processor 631 fails, then further datapackets for file transactions associated with SSD drives 611 can bemanaged by processor 632. These data packets can be received overinterface module 641, or can instead be received over interface module640 since interface module 640 is communicatively coupled to processor632 over at least one PCIe switch 634. To handle incoming data packetsoriginally intended for failed processor 631, processor 632 can inheritnetwork connection states from processor 631, such as TCP/IP connectionsand subsequently handle network traffic originally handled by failedprocessor 631. Failover pathway 691 indicates one example flow of dataand file transactions for SSD drive 611 when processor 631 has failed.

Processor 632 can monitor another processor, such as processor 631,using different methods. In a first example, processor 632 sendsperiodic messages to processor 631 and awaits response messages fromprocessor 631. If the response messages fail to come within apredetermined amount of time, then processor 632 can determine thatprocessor 631 has failed. The periodic messages can include probepackets sent through a network stack or stacks between processors, suchas over a network interface that processor 631 and 632 can communicate.In another example, a set of registers or bits are maintained forprocessor 631 in processor 632 that are set or reset by processor 631.If a register or bit associated with processor 631 has not been properlyset or reset within a predetermined amount of time, then processor 632can determine that processor 631 has failed. Although two processors areshown in FIG. 6 as monitoring each other, in other examples a greaternumber of processors can be included. This greater number of processorscan each be configured be monitored for failure by all other processors,by only one other processor (dual mode), or by two other processors(tri-mode). Any combination of monitored and monitoring processors canbe employed.

Once a processor has been determine to have failed, the monitoringprocessor, such as processor 632, can be configured to initiate arestart, reboot, or power cycle of the failed processor. If the failedprocessor comes back online and functioning after the restart, reboot,or power cycle, then those SSD drives that were transitioned to themonitoring processor can be re-managed by the previously failedprocessor.

FIG. 7 is a system diagram illustrating storage system 700, which can bean example of some elements of system 100 in FIG. 1. FIG. 7 includeselements indicated in FIG. 6 as well as additional elements. Theseadditional elements include storage sleds 710 and 715, processing module730, and interface modules 740-741. Storage sled 710 includes SSD drives711 and PCIe switch 712. Storage sled 715 includes SSD drives 716 andPCIe switch 717. Although four storage sleds are included in FIG. 7, itshould be understood that any number of storage sleds can be included.Also, although four SSD drives are included in each storage sled, itshould be understood that any number of SSD drives can be included. I/Omodule 620 includes PCIe switches 621-622, and I/O module 625 includesPCIe switches 626-627. Although two PCIe switches are shown for each I/Omodule in FIG. 7, it should be understood that any number can beincluded. Processing module 730 includes processors 731-732 and PCIeswitches 733-734, and further elements can be included as discussedabove.

Interface module 740 communicates over at least three interfaces, namelya network interface 742 for TCP/IP, iSCSI, or NVMe traffic, a clusterPCIe interconnect 743 for interconnection between different 3U modules,such as those discussed in FIG. 2, and a processing module PCIeinterface 756. Interface module 741 communicates over at least threeinterfaces, namely a network interface 745 for TCP/IP, iSCSI, or NVMetraffic, a cluster PCIe interconnect 744 for interconnection betweendifferent 3U modules, such as those discussed in FIG. 2, and aprocessing module PCIe interface 757. Processing module 630 communicateswith any of storage sleds 610, 615, 710, and 715 over PCIe links 654 and655 which are switched through associated PCIe switches on associatedI/O modules. Processing module 730 communicates with any of storagesleds 610, 615, 710, and 715 over PCIe links 754 and 755 which areswitched through associated PCIe switches on associated I/O modules.Storage sled 710 communicates with different I/O modules over respectivePCIe links 750-751. Storage sled 715 communicates with different I/Omodules over respective PCIe links 752-753. The elements of FIG. 7 forma storage system with multiple redundant PCIe links between processingmodules and storage sleds by way of I/O modules, such as those discussedherein.

In operation, a specific processor of a processing module will beconfigured to manage specific SSD drives on various storage sleds. Forexample, processor 731 can be configured to manage SSD drives 711 ofstorage sled 710 while processor 732 can be configured to manage SSDdrives 716 of storage sled 715. Other combinations can be used,including a subset of SSD drives of a particular storage sled managed bya processor and the remaining SSD drives of that particular storage sledmanaged by another processor. Managing an SSD drive comprises aprocessor initializing the SSD drive and handling read, write, or otherfile or object transactions for the SSD drive. Each processor canreference a data structure stored on a non-transitory computer readablemedium to determine which SSD drives are to be managed.

However, once a particular processor has been configured to manage aparticular SSD drive, the processor might experience a failure orinterruption in normal operation. Although FIG. 6 focuses on processorson a shared processing module 630, the examples of FIG. 7 discussprocessors that may reside on separate processing modules, namelyprocessing module 630 and processing module 730.

Responsive to the failure of a processor, such as processor 631, anotherprocessor 732 can be configured to take over management of the SSDdrives previously managed by the failed processor 631. Processor 732 caninitialize the SSD drives previously managed by the failed processor 631and begin managing file operations, such as read and write operations,for those SSD drives. Processor 732 can include failover table 780 whichindicates SSD drives for which processor 732 manages in the event of afailure of another processor. Failover table 780 can also indicate aprocessor or processors that processor 732 monitors to determine if anyfailures have occurred.

Table 780 can comprise a listing of various portions of a shared PCIeaddress space that each of the processors of system 700 handles storageoperations for, such as an associated address range. This address rangecorresponds to one or more of the SSDs on the storage sleds andindicates which SSDs are managed by a particular processor. Eachprocessor can include table 780 which indicates which SSDs are managedby that processor and which SSDs are managed by the processors whichthat processor monitors for failures. Additionally, in load-balancingexamples, table 780 can include all of the various PCIe address rangesfor all processors, so that when a particular processor receives astorage operation that particular processor can check the associatedPCIe address against table 780 to determine if that particular processorshould manage the storage operation or if the storage operation shouldbe transferred to another of the processors for further handling. FIG.10 shows an example PCIe address space that is subdivided among variousprocessors which can manage various ones of the SSDs in a storagesystem.

In FIG. 7, different pathways for data and other file transactions areshown. In a first pathway, namely normal pathway 790, interface module640 receives data packets corresponding to a file transaction (such asdata store or retrieve transactions) and transfers the data packets overPCIe switch 634 for delivery to processor 631. Processor 631 determinesthat it “owns” SSD drives that correspond to the file transaction andproceeds to handle the file transaction for the particular SSD drives,such as SSD drives 611. If processor 631 fails, then further datapackets for file transactions associated with SSD drives 611 can bemanaged by processor 732. These data packets can be received overinterface module 741 when processor 732 begins accepting packetsoriginally accepted by processor 631. Alternatively, these data packetscan be received over interface module 640 and transferred for deliveryto processor 732 over a PCIe switch, such as PCIe switch 634, amongothers. To handle incoming data packets originally intended for failedprocessor 631, processor 732 can inherit network connection states fromprocessor 631, such as TCP/IP connections and subsequently handlenetwork traffic originally handled by failed processor 631. Failoverpathway 791 indicates one example flow of data and file transactions forSSD drive 611 when processor 631 has failed.

Processor 732 can monitor another processor, such as processor 631,using different methods. In a first example, processor 732 sendsperiodic messages to processor 631 and awaits response messages fromprocessor 631. If the response messages fail to come within apredetermined amount of time, then processor 732 can determine thatprocessor 631 has failed. The periodic messages can include probepackets sent through a network stack or stacks between processors, suchas over a network interface that processor 631 and 732 can communicate.In another example, a set of registers or bits are maintained forprocessor 631 in processor 732 that are set or reset by processor 631.If a register or bit associated with processor 631 has not been properlyset or reset within a predetermined amount of time, then processor 732can determine that processor 631 has failed. Although two processors areshown in FIG. 7 as monitoring each other, in other examples a greaternumber of processors can be included. This greater number of processorscan each be configured be monitored for failure by all other processors,by only one other processor (dual mode), or by two other processors(tri-mode). Any combination of monitored and monitoring processors canbe employed.

Once a processor has been determine to have failed, the monitoringprocessor, such as processor 732, can be configured to initiate arestart, reboot, or power cycle of the failed processor. If the failedprocessor comes back online and functioning after the restart, reboot,or power cycle, then those SSD drives that were transitioned to themonitoring processor can be re-managed by the previously failedprocessor.

The storage systems described herein include a switch fabric which iscomprised of a plurality of PCIe switch elements along with associatedPCIe links. This switch fabric can include any of the PCIe switchesdiscussed herein, and can be partitioned into a front end data plane anda back end data plane. The front end data plane is employed to transferfile operations, object operations, inter-processor communications,among other control information between processors, processing modules,and network interfaces. The back end data plane is employed to transferdata to and from storage sleds. When multiple processors are employed,the switch fabric is configured with a failover path. If thecommunications goes down for a particular processor, the hardware canautomatically switch PCIe connections over to another processor.Additionally, there is a management processor on each I/O module thatcan be instructed to dynamically change the PCIe paths as necessary byany of the processing modules. The same type of mechanisms that areemployed for failover are used to form a redundant path for each of thecomponents the fabric.

In addition to the paths the hardware provides, there are alsosoftware-based additional layers of redundancy via front end and backend routing. All processors within a storage system are aware of thecurrent state of the hardware and can determine additional routes tostorage sleds as required. This is achieved in part through routerequests locally within the system all the way to rerouting trafficexternally by reverse ARP. Hardware provides minimal latency reroutesfor failover and redundancy. While software increase the levels ofredundant paths that can be changed too. (i.e. when a hardware pathfails software can set up the next path of redundancy for anotherfailure event). There is a table included in each processor that tellsit which SSD it owns normally and which it supports when there is afailover situation. In the event of a failover an interrupt is generatedon the PCIe bus as it switches over. A discovery process of drive(s)assigned to the failed processor is initiated. This is followed by theprocessor populating discovered SSD(s) that match the failover tableinto its active table.

The redundancy and failover are controlled by a couple of mechanisms. Atbootup, each processing node (running a separate instance of anoperating system) gets a configuration that specifies the peer-nodemonitoring algorithm (all nodes monitor all others, dual nodemonitoring, where each node has one peer, and tri-node monitoring whereeach node has two peers monitoring it). The peer algorithm uses ahardware register heartbeat mechanism that each node sets/clears like analternately at a configurable interval (seconds). This lets a processorknow when the peer is crashed and/or loses power. In addition there areSW algorithms running on each side that sends a “probe” packet over thenetwork stack at configurable interval (seconds). After a loss of somemax number of missed probe responses, the peers can determine if thepeer kernel, driver, or operating system software is hung up or crashed.Thus, there is hardware and software checking for enhanced peer statedetermination.

Once a node is determined to be crashed and/or non-responding, afailover event is initiated by one of the peers configured to monitorthe crashed node. This entails two actions: 1) take over ownership theSSDs of the node, 2) initiate a failback operation on a node byrecycling its power or resetting it to reboot. The SSD ownership changeoccurs by one of the monitoring peer's searching a table/array ofdevices owned by the failed node. Note, each peer knows all the devicesowned by peers. The exchange this information periodically (i.e. every60 seconds roughly) and initially when a peer boots and joins thecluster. The peer then executes a userspace script/command thatreconfigures the PCIe switch fabric chip to make the device “visible”through the fabric to the node that wants to obtain ownership. The peerthen checks the SSD's device driver's status register, determines thestate of the SSD/driver, then issues a driver/device reset such that thedevice gets recycled and rediscovered anew (by the monitoring peer thatwants to obtain ownership). The drive now “appears” to the new owningpeer, gets configured and comes on line accessible now via the peer's IPaddress.

In parallel to what is described above, an operating system backendsoftware invalidates layer-2 (Ethernet) routing table entries pointingto the crashed node and generates new layer-2 (Ethernet) routing tablesentries that point to the new owning peer as the destination for storagetransactions that access the SSDs/LBAs contained on the transferredSSDs. These tables are propagated to all the nodes in the cluster.Incoming storage transactions now are routed correctly at layer-2 to thenew owning node. The crashed node is then rebooted by the monitoringpeer. It will continue attempting to reset it until it comes back andre-joins the cluster.

FIG. 8 is a system diagram illustrating a storage system, which can bean example of some elements of system 100 in FIG. 1. FIG. 8 includeselements indicated in FIGS. 6 and 7 as well as additional elements.These additional elements include PCIe switches 830-833. I/O module 620includes PCIe switches 621-622 and 832, and I/O module 625 includes PCIeswitches 626-627 and 833. Processor 632 communicates over PCIe link 840with PCIe switch 830. Processor 631 can also communicate with PCIeswitch 830. Processor 732 communicates over link 841 with PCIe switch831. Processor 731 can also communicate with PCIe switch 831. PCIeswitch 830 communicates with both PCIe switch 832 and 833 overassociated PCIe links 842 and 843. PCIe switch 831 communicates withboth PCIe switch 832 and 833 over associated PCIe links 844 and 845.

FIG. 8 illustrates front end data plane cross connection betweenprocessing modules 630 and 730 over any of I/O module 620 and 625. Asdiscussed above for FIGS. 6 and 7, a particular processor can manage aparticular I/O sled or set of SSD drives. However, data packets relatedto data transactions for a particular SSD drive can be received over anyof interface modules 640, 641, 740, and 741. This configuration ofmultiple interface modules each communicating over various TCP/IP linksallows for load balancing, highly parallelized receipt of transactionsfrom external systems, and reception of traffic for any SSD drive overany network interface included in system 800. A particular processortypically manages traffic for a particular SSD on the back end dataplane side, such as for read and write transactions to a particularstorage sled, but file transactions can be received over a networkinterface by any processor. FIG. 8 illustrates one example process oftransferring a file transaction received at one processor to anotherprocessor which manages the storage elements associated with the filetransaction. The operations described in FIG. 8 are further detailed inFIGS. 9A and 9B, which will be discussed below.

In FIG. 8, a file transaction is received over link 645 by interfacemodule 641, as indicated by “transaction receive 892” in FIG. 8. Thisfile transaction can include a file or object operation or transaction,such as read, write, delete, modify, or other file system operations fordata. In this example, the affected data is stored on SSD drive 716 onstorage sled 715. However, the file transaction, in the form of one ormore packets, is received by interface module 641 and transferredinitially to processor 632 by way of PCIe switch 634. Transaction 892can indicate a storage address for which the data associated withtransaction 892 is to be written to or read from, among otheroperations.

Processor 632 includes routing table 880 which comprises one or moredata structures that indicate which particular processor of system 800manages data for the storage address space contained within the storagesleds of system 800. As discussed above, a particular processor managesa particular set of SSD drives or storage sleds. However, filetransactions can be received over any network interface by anyprocessor. Each processor includes a routing table or other datastructure that indicates which processor manages a particular addressspace.

Table 880 can comprise a listing of various portions of a shared PCIeaddress space that each of the processors of system 800 handles storageoperations for, such as an associated address range. This address rangecorresponds to one or more of the SSDs on the storage sleds andindicates which SSDs are managed by a particular processor. Eachprocessor can include table 880 which indicates which SSDs are managedby that processor and which SSDs are managed by the processors whichthat processor monitors for failures. Additionally, in load-balancingexamples, table 880 can include all of the various PCIe address rangesfor all processors, so that when a particular processor receives astorage operation that particular processor can check the associatedPCIe address against table 880 to determine if that particular processorshould manage the storage operation or if the storage operation shouldbe transferred to another of the processors for further handling. FIG.10 shows an example PCIe address space that is subdivided among variousprocessors which can manage various ones of the SSDs in a storagesystem.

In this example, routing table 880 indicates that processor 732 managesthe storage devices associated with transaction 892. Processor 632transfers transaction 892 over at least PCIe switch 830, PCIe switch833, and PCIe switch 831 for delivery to processor 732, as indicated by“transaction transfer 893” in FIG. 8. Processor 732 then handlestransaction 892. If transaction 892 is a write transaction, thenprocessor 732 writes data associated with transaction 892 into anassociated SSD drive, such as SSD drive 716 of storage sled 715, asindicated by “transaction commit 894” in FIG. 8. For read transactions,processor 732 reads associated data from an SSD drive and transfers theread data for delivery to an external system that initiated thetransaction. This read data can be transferred over a network interfaceassociated with processor 732, such as interface module 741.

In iSCSI examples, FIG. 8 can incorporate features for iSCSI portalgroups. For example, transferring an iSCSI transaction received at afirst processing system to another processing system in the same portalgroup can be achieved by the operation described in FIG. 8. FIG. 8illustrates PCIe front end data plane cross connection betweenprocessing modules 630 and 730 over any of I/O module 620 and 625. Asdiscussed above for FIG. 1, a particular processor can manage aparticular I/O sled or set of SSD drives. However, data packets relatedto storage transactions for a particular SSD drive can be received overany of interface modules 640, 641, 740, and 741. This configuration ofmultiple interface modules each communicating over various TCP/IP linksallows for load balancing, highly parallelized receipt of transactionsfrom external systems, and reception of traffic for any SSD drive overany network interface included in system 800. A particular processortypically manages traffic for a particular SSD on the back end dataplane side, such as for read and write transactions to a particularstorage sled, but storage transactions can be received over a networkinterface by any processor.

In the example shown in FIG. 8, an iSCSI portal group is discussed asrelated to receiving an iSCSI storage transaction over a first networkinterface associated with a first processor and that processortransferring the storage transaction for handling by a second processor.In FIG. 8, an iSCSI portal group is associated with at least links 642,645, 742, and 745, which in turn are associated with correlated ones ofprocessors 631, 632, 731, and 732. Each of the interface modulescomprises at least one network interface link of a network interfacecard, and has a different associated TCP/IP address associatedtherewith. These various TCP/IP addresses are grouped into an iSCSItarget portal group, as shown in FIG. 8.

However, in FIG. 8, each processor 631, 632, 731, 732, only handles aparticular subset of the total number of SSD drives in system 800. Forexample, processor 632 might only manage SSD drives on storage sled 615and processor 732 might only manage SSD drives on storage sled 715.Other storage drive and sled configurations are possible. Thus, whenprocessor 632 receives a storage transaction for the portal group ofsystem 800, the storage transaction is first transferred to a managingprocessor, namely processor 732 in this example. Processor 632references routing table 880 to determine that processor 732 manages theSSD drives associated with the storage transaction. The storagetransaction can identify an address or address range for the storagetransaction, and this address or address range can be used by processor632 to determine that processor 732 should handle the transactionfurther. Responsively, processor 632 transfers the storage transactionfor delivery to processor 732 over a front end data plane PCIe fabric.This is shown in FIG. 8 as operation 893. The front end data plane PCIefabric is comprised of a plurality of PCIe crosspoint switches, namelyPCIe switches 634, 734, and 830-833, along with any associated PCIelinks. Other pathways can be employed. Further discussion of iSCSIportal groups is found in FIGS. 9A and 12.

As one example or transferring transactions received by one processor orprocessing system for handling by another processor or processingsystem, FIG. 9A is presented. FIG. 9A illustrates storage system 900communicating over an iSCSI protocol. Storage system 900 includesprocessing system 910, processing system 920, and PCIe switch 930.Further storage system elements such as found in the previous figurescan be included in system 900, but are omitted for clarity in FIG. 9A.In alternative examples, NVMe protocol handling is discussed below inFIG. 9B.

In FIG. 9A, processing system 910 and processing system 920 arecommunicatively coupled by at least PCIe switch 930. Further redundantPCIe switches can be employed. PCIe switch 930 is included in a frontend data plane of a storage system, and allows for transfer of filetransactions between processors.

In operation, packets 901 are received by processing system 910. Thesepackets can be received by a network interface card associated withprocessing system 910, such as over an Ethernet interface of processingsystem 910. Packets 901 are associated with a storage transaction, suchas a write transaction or read transaction, for data storage on anassociated storage drive, such as a storage sled or SSD as discussedherein. Each of processing systems 910 and 920 include networkinterfaces that can receive storage transactions for each other forredundancy, load balancing, failover protection, among other features.Additionally, as discussed above, each processing system manages or ownsstorage drives that are particular to each processing system.Specifically, processing system 910 will manage a first set of storagedrives, while processing system 920 will manage a different second setof storage drives. However, if an iSCSI storage operation for the secondset of storage drives is received over a network interface associatedwith a processing system that manages the first set of storage drives,then the operations detailed in FIG. 9A is followed.

In FIG. 9A, packets 901 are associated with a storage operation forstorage drives managed by processing system 920. However, packets 901might be received by processing system 910 which might manage adifferent set of storage drives not associated with packets 901. Thesepackets will be received into the various network stack layers of anetwork interface of processing system 910, such as PHY layer 1 911, andETH layer 2 912. If packets 901 were for a storage drive managed byprocessing system 910, then these packets can proceed further into IPlayer 3 913 or further layers 914-916 for storage as data 981. However,if packets 901 were for a storage drive managed by another processingsystem, such as processing system 920, then the process detailed belowcan allow for processing system 920 to handle packets 901 for storage asdata 982.

Specifically, once packets 901 are stored in buffer 912B, processingsystem 910 executes process 912P to determine if packets 901 are relatedto a storage transaction for storage drives managed by processing system910 or by another processing system. Process 912P includes checkingvarious headers and field values of packets 901 to determine if packets901 are associated with storage drives that processing system 910manages or another processing system. FIG. 9A illustrates one exampleprocess 912P.

First, an Ethernet header 902 of packets 901 are examined to determine avalue of the Ethertype field. If the Ethertype field indicates an IPpacket, then IP header 903 is examined. If the protocol field in IPheader 903 indicates TCP and the destination port identifier indicatesan iSCSI port, then iSCSI header 904 is examined. iSCSI header 904indicates various properties including an iSCSI target identifier and alogical block address (LBA) associated with the iSCSI target identifier.This target identifier and logical block address can be used todetermine if the storage transaction comprising packets 901 is for thelocal processor/processing system 910 or a remote processor/processingsystem 920, among other processors/processing systems.

Table 990 indicates an example routing configuration to determine whichprocessing system is to handle incoming iSCSI packets. It should beunderstood that the values indicated for table 990 are merely exemplary,and other values can be used in practice. Table 990 indicates threecolumns, with the first two corresponding to the target ID and the LBAfound in iSCSI header 904. The third column corresponds to a processingnode identifier. The processing node ID is correlated to a PCIe addressrange, with each processing system or processor in a storage system thatshares a front end PCIe fabric having a corresponding address range. Ifthe address range is for itself (local processing system 910), thenprocessing system 910 determines that it should handle packets 901through layers 913-916 and storage of corresponding data 981. If theaddress range is for a remote processing system, such as processingsystem 920, then processing system 910 determines that it shouldtransfer packets 901 to the remote processing system.

FIG. 9A also illustrates transfer of packets 901 to another processingsystem. If packets 901 are to be handed by another processing system,such as determined using routing table 990 discussed above, thenprocessing system 910 transfers packets 901 from buffer 912B to the PCIememory address location indicated by table 990 as the processing nodeID. FIG. 9A shows processing system 910 transferring packets 901 over aPCIe fabric comprised of at least PCIe switch 930 into a buffer ofprocessing system 920, namely buffer 922B. Processing system 910 canalso issue an interrupt or other signal to processing system 920 whichindicates that packets 901 are waiting in buffer 922B for handling byprocessing system 920.

Processing system 920 checks buffer 922B responsive to an interruptissued by processing system 910, or due to periodic polling of buffer922B. This buffer 922B can be a buffer maintained by ETH layer 2 922 ofprocessing system 920. ETH layer 2 922 can have more than one bufferassociated therewith. A first buffer can be a circular buffer used forreceiving packets over PHY layer 1 921. A second buffer can be a PCIebuffer, such as buffer 922B, that accepts packets from other processingsystems over a PCIe fabric.

In this example, buffer 922B is associated with ETH layer 2 922 of anetwork stack of processing system 920. Processing system 920 thenprocesses packets 901 as if processing system 920 itself had receivedpackets 901 over its own network interface, such as by passing packets901 to a higher layer in the network stack, including IP layer 3 923.These packets can then be handled by processing system 920 for theparticular storage transaction that are indicated by packets 901. Forexample, if packets 901 describe a write operation for SSDs managed byprocessing system 920, then data 982 is transferred for storage inassociated storage drives by processing system 920.

Since packets 901 were originally addressed to processing system 910,portions of packets 901 can be transformed by processing system 920 tobe handled by a network stack of processing system 920. The recipientpeer/remote node, namely processing system 920, receives packets 901 andtransforms portions of packets 901, such as TCP frames, and re-buildsthe Ethernet header, the IP header to appear as if it was sent directlyto the servicing node to prevent a network stack of processing system920 from rejecting packets 901. This transformation can includemodifying an associated MAC addresses of packets 901, Ethernet CRCvalue, destination IP address, and other header properties. In otherexamples, processing system 920 does not transform the portions ofpackets 901 an instead inhibits rejection of packets 901 due to MACaddress, Ethernet CRC value, or destination IP address mismatches.

In this manner, processing system 910 can receive packets for storagetransactions for storage drives handled by another processing system920. Processing system 910 inspects packet 901 and references table 990to determine if to transfer packets 901 over the PCIe fabric to thecorrect processing system. Moreover, the packets are transferred into alayer 2 buffer of the correct processing system and the correctprocessing system handles the packets as if it had originally receivedthe packets instead of the processing system that instead received thepackets over a different network interface.

In specific examples, network driver software of a second processingsystem is modified to intake packets received at a first processingsystem as if it was received at that second processing system. Likewise,the driver software of the first processing system is also modified totransfer packets received by the first processing system for deliveryinto a network stack of the second processing system. The layer 2 driversoftware of the second processing system can receive an interruptgenerated by the first processing system to indicate that new packetsare waiting in an associated layer 2 buffer of the second processingsystem.

As another example or transferring storage transactions received by oneprocessor or processing system for handling by another processor orprocessing system, FIG. 9B is presented. FIG. 9B illustrates storagesystem 900 communicating using an NVMe protocol. Storage system 900includes processing system 910, processing system 920, and PCIe switch930. Further storage system elements such as found in the previousfigures can be included in system 900, but are omitted for clarity inFIG. 9B.

In FIG. 9B, processing system 910 and processing system 920 arecommunicatively coupled by at least PCIe switch 930. Further redundantPCIe switches can be employed. PCIe switch 930 is included in a frontend data plane of a storage system, and allows for transfer of filetransactions between processors.

In operation, frames 940 are received by processing system 910. Theseframes can be received by an NVMe interface associated with processingsystem 910, such as over an NVMe interface card coupled to processingsystem 910. Frames 940 are associated with a storage transaction, suchas a write transaction or read transaction, for data storage on anassociated storage drive, such as a storage sled or SSD as discussedherein. Frames 940 can comprise one or more NVMe frames, and can bebundled into one or more packets. Each of processing systems 910 and 920include communication interfaces that can receive storage transactionsfor each other for redundancy, load balancing, failover protection,among other features. Additionally, as discussed above, each processingsystem manages or owns storage drives that are particular to eachprocessing system. Specifically, processing system 910 will manage afirst set of storage drives, while processing system 920 will manage adifferent second set of storage drives. However, if an NVMe storageoperation for the second set of storage drives is received over an NVMeinterface associated with a processing system that manages the first setof storage drives, then the operations detailed in FIG. 9B is followed.

In FIG. 9B, frames 940 are associated with an NVMe storage operation forstorage drives managed by processing system 920. However, frames 940might be received by processing system 910 which might manage adifferent set of storage drives not associated with frames 940. Theseframes will be received into the various protocol stack layers of anNVMe interface of processing system 910, such as PHY layer 1 917, andNVMe layer 2 918. If frames 940 were for a storage drive managed byprocessing system 910, then these frames can proceed further for storageas data 983. However, if frames 940 were for a storage drive managed byanother processing system, such as processing system 920, then theprocess detailed below can allow for processing system 920 to handleframes 940 for storage as data 984.

Specifically, once frames 940 are stored in buffer 918B, processingsystem 910 executes process 918P to determine if frames 940 are relatedto a storage transaction for storage drives managed by processing system910 or by another processing system. Process 918P includes checkingvarious headers and field values of frames 940 to determine if frames940 are associated with storage drives that processing system 910manages or another processing system. FIG. 9B illustrates one exampleprocess 918P.

First, an NVMe header 942 of frames 940 are examined. NVMe header 942indicates various properties including an NVMe logical unit number(VLUN) and possibly a logical block address (LBA) associated with theVLUN. This VLUN and LBA information can be used to determine if thestorage transaction comprising frames 940 is for the localprocessor/processing system 910 or a remote processor/processing system920, among other processors/processing systems.

Table 991 indicates an example routing configuration to determine whichprocessing system is to handle incoming NVMe frames. It should beunderstood that the values indicated for table 991 are merely exemplary,and other values can be used in practice. Table 991 indicates threecolumns, with the first two corresponding to the VLUN and the LBA foundin NVMe header 942. The third column corresponds to a processing nodeidentifier. The processing node ID is correlated to a PCIe addressrange, with each processing system or processor in a storage system thatshares a front end PCIe fabric having a corresponding address range. Ifthe address range is for itself (local processing system 910), thenprocessing system 910 determines that it should handle frames 940 andstorage of corresponding data 983. If the address range is for a remoteprocessing system, such as processing system 920, then processing system910 determines that it should transfer frames 940 to the remoteprocessing system.

FIG. 9B also illustrates transfer of frames 940 to another processingsystem. If frames 940 are to be handed by another processing system,such as determined using routing table 991 discussed above, thenprocessing system 910 transfers frames 940 from buffer 918B to the PCIememory address location indicated by table 991 as the processing nodeID. FIG. 9B shows processing system 910 transferring frames 940 over aPCIe fabric comprised of at least PCIe switch 930 into a buffer ofprocessing system 920, namely buffer 928B. Processing system 910 canalso issue an interrupt or other signal to processing system 920 whichindicates that frames 940 are waiting in buffer 928B for handling byprocessing system 920.

Processing system 920 checks buffer 928B responsive to an interruptissued by processing system 910, or due to periodic polling of buffer928B. This buffer 928B can be a buffer maintained by NVMe layer 2 928 ofprocessing system 920. NVMe layer 2 928 can have more than one bufferassociated therewith. A first buffer can be a buffer used for receivingNVMe frames over PHY layer 1 927. A second buffer can be a PCIe buffer,such as buffer 928B, that accepts frames from other processing systemsover a PCIe fabric.

In this example, buffer 928B is associated with NVMe layer 2 928 ofprocessing system 920. Processing system 920 then processes frames 940as if processing system 920 itself had received frames 940 over its ownNVMe interface. These packets can then be handled by processing system920 for the particular storage transaction that are indicated by frames940. For example, if frames 940 describe a write operation for SSDsmanaged by processing system 920, then data 984 is transferred forstorage in associated storage drives by processing system 920.

Since frames 940 were originally received by processing system 910,portions of frames 940 can be transformed by processing system 920 to behandled by a protocol stack of processing system 920. The recipientpeer/remote node, namely processing system 920, receives frames 940 andtransforms portions of frames 940, such as NVMe frames, and re-buildsthe NVMe header to appear as if it was sent directly to the servicingnode to prevent an NVMe stack of processing system 920 from rejectingframes 940. This transformation can include modifying an associated MACaddresses of frames 940, CRC values, destination network addresses, andother header properties. In other examples, processing system 920 doesnot transform the portions of frames 940 an instead inhibits rejectionof frames 940 due to MAC address, CRC value, or destination networkaddress mismatches.

In this manner, processing system 910 can receive NVMe frames forstorage drives handled by another processing system 920. Processingsystem 910 inspects packet 940 and references table 991 to determine ifto transfer frames 940 over the PCIe fabric to the correct processingsystem. Moreover, the frames are transferred into an NVMe layer 2 bufferof the correct processing system and the correct processing systemhandles the frames as if it had originally received the frames insteadof the processing system that instead received the frames over adifferent network interface.

In specific examples, network driver software of a second processingsystem is modified to intake frames received at a first processingsystem as if it was received at that second processing system. Likewise,the driver software of the first processing system is also modified totransfer frames received by the first processing system for deliveryinto a network stack of the second processing system. The layer 2 driversoftware of the second processing system can receive an interruptgenerated by the first processing system to indicate that new frames arewaiting in an associated layer 2 buffer of the second processing system.

FIG. 10 is a diagram illustrating PCIe address space 1000, which can beemployed for a PCIe fabric as discussed herein. PCIe address space 1000is a 2⁶⁴ bit address space associated with the PCIe address space of aPCIe switch fabric of a storage system. PCIe address space 1000 mapsvarious ranges to buffers of particular processors or processingsystems. Specifically, an address range is associated with a processingsystem, such as processing system 1 through N, among others. In astorage system, such that described in FIG. 1, a particular processorwill manage and instantiate a particular subset of the total number ofstorage drives, but storage transactions can be received into anyprocessor over an associated network interface. To enable load balancingand redundancy, each processor or processing system can include anassociated network interface that can receive packets for storagetransactions, including read and write transactions. If the packetsreceived at a first processor are for storage drives not managed by thefirst processor, then these packets are transferred for handling byanother processor.

To facilitate the transfer of the packets and associated data from oneprocessor to another processor, PCIe address space 1000 is subdividedamong the various processing systems. Each processing system in astorage system is assigned a particular quantity of storage drives, andthus is configured to intake packets and data addressed to a particularPCIe address range. A first processor transferring packets for deliveryto another processor does not need to have knowledge of which exactprocessor is configured to handle the storage drives for the packets.Instead, the storage space of the entire storage system is divided amongthe various processors in the storage system, and each individualprocessor is configured to respond to only transactions directed to aspecific address range of the PCIe address space. Thus, a storagesystem, such as storage system 100 in FIG. 1, can have a total storagespace of 2⁶⁴ bytes, or 16 exbibytes of byte-addressable memory.

When a first processor receives data packets for a storage operation,such as a read or write operation, the first processor determines if thestorage address included in the data packets is for an address rangemanaged by the first processor. If the data packets indicate a storageaddress managed by the first processor, then the first processor handlesthe storage operation, such as by transferring write data for storage instorage drives managed by the first processor or by initiating a readoperation for storage drives managed by the first processor. If the datapackets indicate a storage address managed by the second processor, thenthe first processor merely transfers the data packets to the address bythe mapping to PCIe address space 1000. The particular processorconfigured to handle storage operations for those storage addressesindicated in the data packets will be configured to receive any datatransferred to the associated PCIe address range, and responsivelyhandle the storage operation for the storage drives managed by thatparticular processor. The first processor does not need to know whichparticular processor manages particular storage drives, and insteadmerely transfers the data packets for a storage transaction to aparticular PCIe address which is automatically routed to the correctprocessor for further handling. From the viewpoint of the processorwhich receives the packets, the PCIe address space includes addressranges that it manages and address ranges it does not manage.

Moreover, each processor can have its address space sub-divided intobuffer spaces for the other processors of the system. For example,processor 1 might have a first address range, but when processor 2transfers data to processor 1, processor 2 uses its own identity as ahash into the address space of processor 1. In this manner, eachprocessor's address space is segregated into sub-spaces that are eachassociated with another processor in the storage system.

In FIG. 10, PCIe address space 1000 is subdivided among the total numberof processors in a storage system, such as in storage system 100 ofFIG. 1. For example, if a storage system has 12 processors, then PCIeaddress space 1000 can be subdivided among the 12 processors. Otherconfigurations are possible. PCIe address space 1000 can be equallysubdivided, unequally subdivided, or a portion of PCIe address space1000 can be subdivided, among other configurations. The total amount ofPCIe address space utilized by a storage system can correspond to thetotal addressable storage space provided by the collection of storagedrives employed. For example, if 2 terabytes of storage space isemployed, then PCIe address space 1000 can have a maximum rangecorresponding to the 2 terabytes of storage space and each processor canhave a corresponding sub-portion of that 2 terabytes of storage spaceallocated thereto. In yet other examples, the total amount of PCIeaddress space might not correspond to the total storage space of thestorage system, and might be a smaller amount and used just forshuttling storage packets among the various processors for loadbalancing and failover fault handling. Of course, the load balancing andfailover features can be employed in the example where PCIe addressspace 1000 is correlated to the total addressable storage space of thestorage system.

For each processor address space in PCIe address space 1000, a furthersubdivision is performed. Specifically, processor 0 and processor 1address spaces 1010-1011 are shown in FIG. 10. Processor 0 address space1010 includes portions assigned to each other processor in a storagesystem, namely processor 1-N. Additionally, processor 0 address space1010 includes a miscellaneous area that includes space for lockindicators, flags, and interrupts. These interrupts can be used tosignal a receiving processor that data or packets are ready forhandling. The lock and flag spaces can be used for handshaking,semaphoring, or other uses during data or packet transfer betweenprocessors.

In operation, if processor 0 desires to transfer packets to processor 1,then processor 0 will transfer the packets into the space allocated forprocessor 0 within address space 1011 of processor 1. These packets canbe transferred over an associated PCIe fabric. Any appropriate interruptor flags can be set by processor 0 in the interrupt/flag/lock space ofprocessor 1 address space 1011. Processor 1 can detect that packets orother data is waiting in the address space for processor 0 and actaccordingly, such as by initiating a storage transaction indicated bythe packets transferred by processor 0 into the address space ofprocessor 1.

As discussed herein, each processor of a storage system is configured tohave visibility to all storage drives of a particular storage system,such as by using the memory mapped address spaces in FIG. 10. However,each processor does not manage all storage drives of a particularstorage system, and instead only manages a subset of the total storagedrives. The PCIe front end data plane is employed to exchange data andpackets between processors and maintain visibility for each processor toeach storage drive. A PCIe back end data plane is employed to transferdata to and from actual storage drives and the various managingprocessors. One or more PCIe switches form the PCIe back end data plane.An initialization process is performed for each processor of a storagesystem to allow each processor to manage the particular subset of thestorage drives. Typically for a processor that manages storage drives,the processor performs a PCIe discovery process to find the storagedrives connected via PCIe interfaces to the processor to initialize orenumerate those storage drives. In some examples, for storage drives notmanaged by a processor, initialization software spoofs the enumerationprocess for that processor so that processor thinks that the PCIediscovery process has found all storage drives in the storage system.

For example, a first processor, upon boot up or responsive to areconfiguration process, performs a scan of any associated PCIeinterfaces for devices that the first processor is to manage andenumerate. However, in a storage system such as that included in FIG. 1,many storage drives are included, and only a particular processormanages a subset of the total storage drives. The first processor isallowed to enumerate and initialize a subset of the storage drives thatit is to manage. Similarly, other processors will enumerate andinitialize different subsets of the storage drives. In this manner, eachprocessor can access all storage drives via the PCIe front end dataplane, but only physically manages a subset of the storage drives on thePCIe back end data plane. Thus, a large shared storage space can beprovided using many different processors, with each processor onlymanaging a small associated portion of the total storage space.

Normally, each processor will want to enumerate and initialize all thestorage drives that each processor “sees” over the PCIe bus, which wouldinclude all the storage drives of a storage system since all storagedrives are connected to all processors via a shared back end PCIe switchfabric, such as found in FIG. 1. By segregating a back end data planeinitialization process for each processor, the storage drives can beallocated among the various processors even though a shared front enddata plane PCIe address space is employed across all processors.

When a particular processor fails, then the storage drives managed bythat particular processor can be re-assigned to another processor formanaging storage transactions. An initialization process can beperformed between the storage drives associated with the failedprocessor and the replacement processor to enumerate those storagedrives with the replacement processor. The replacement processor isconfigured to handle transactions addressed to the failed processor'saddress space, in addition to any address spaces the replacementprocessor might have previously managed.

As another example of presenting one or more NVMe targets over a PCIeinterface, FIG. 11 is presented. FIG. 11 is a system diagramillustrating storage system 1100. Storage system 1100 includes storagespace 1101 which comprises a plurality of storage drives contained onstorage sleds, such as the storage drives on the storage sleds ofFIG. 1. Storage space 1101 can be an example of a storage space definedover one or more of storage sleds 110 of FIG. 1, although otherconfigurations are possible. For example, storage space 1101 can bestriped, spanned, mirrored, or parallelized over more than one storagesled 110. Storage sleds 110 of FIG. 1 can include more than one storagespace. In this example, however, storage space 1101 is discussed.Storage space 1101 includes three logical storage units, such as logicalstorage units 1102-1104. These logical units can define a storage unitsuch as a VLUN, logical disk, among other logical storage arrangements.In this example, however, logical storage units 1102-1104 each representan NVMe VLUN.

Turning now to the elements of FIG. 11, storage system 1100 alsoincludes at least one processing system, namely processing system 1110.More than one processing system can be included, such as found inFIG. 1. Processing system 1110 includes at least one processor 1111coupled to an associated PCIe interface 1112. PCIe interface 1112 can becoupled over an associated PCIe connectors 1114 or PCIe switch 1113 toexternal systems, such as servers 1130, 1140, and 1150. Processingsystem 1110 can also communicate with storage drives that comprisestorage space 1101 over PCIe switch 1114. PCIe switch 1113 can comprisea front end data plane PCIe fabric, and PCIe switch 1114 can comprise aback end data plane PCIe fabric, although unified PCIe fabrics can beemployed. PCIe switches 1113-1114 each comprise PCIe crosspoint switchesthat allow communication among the various elements of FIG. 11.

In operation, processing system 1114 can receive storage transactionsfrom external systems over an external PCIe interface. Processing system1114 presents the storage resources of storage system 1100 as one ormore NVMe VLUNs. In the example of FIG. 11, a first VLUN is presented asVLUN 1121, a second VLUN is presented as VLUN 1122, and a third VLUN ispresented as VLUN 1123.

NVMe emulation software 1120 is shown as performing the presentation andemulation of NVMe VLUNs over the PCIe interfaces of FIG. 11. NVMeemulation software 1120 can include an NVMe abstraction layer withadvanced virtualization features to present one or more NVMe targetsover one or more PCIe physical interfaces. NVMe emulation software 1120includes generation and maintenance. NVMe emulation software 1120includes advanced transaction queuing, Message Signaled Interrupts (MSI)for PCI (MSI-X) interrupt generation, error reporting, single rootvirtualization and sharing (SR-IOV), name spaces, hinting, powermanagement features, and security features, among other features.

These VLUNs can be presented as an NVMe target, such as NVMe target A orNVMe target B in FIG. 1. An NVMe target can present the storageresources of storage system 1100 as a storage target, such as emulatinga single storage drive, over a PCIe interface. In this manner, aplurality of storage drives that comprise storage space 1101 can bepresented as a single NVMe target to an external system over PCIeinterfaces. Processing system 1110 can receive NVMe storagetransactions, such as NVMe frames, and handle these storage transactionsfor storage and retrieval of data within storage space 1101.

In a specific example, server 1130 is coupled over PCIe link 1161 usingPCIe signaling with processing system 1110 over PCIe connectors 1115. Acable carrying PCIe signaling, such as a PCIe direct connect cable canbe employed. Logical storage unit 1102 is presented as NVMe VLUN 1121 toserver 1130 over PCIe link 1161. Server 1130 can mount this NVMe VLUN1121 as a logical storage unit and use this storage space as a massstorage device for storing and retrieving data. Server 1130 can issuestorage transactions, such as read and write transactions, over PCIelink 1161. These storage transactions can include one or more NVMeframes 1131 that indicate an address for data to be read or written (andmay include data 1132 for write transactions). The address can include alogical block address or object identifier, among other addresses ordata identifiers. Processing system 1110 receives these NVMe frames overPCIe link 1161 and handles the associated storage transactions. In awrite example, data 1132 is stored in storage space 1101, specificallyin logical storage unit 1102. Data 1132 can be striped across manystorage drives or parallelized for storage among many storage drivesthat comprise storage space 1101.

In a second example, server 1140 is coupled over PCIe link 1162 usingPCIe signaling with processing system 1110 over PCIe connectors 1115. Acable carrying PCIe signaling, such as a PCIe direct connect cable canbe employed. Logical storage unit 1103 is presented as NVMe VLUN 1122 toserver 1140 over PCIe link 1162. Server 1140 can mount this NVMe VLUN1122 as a logical storage unit and use this storage space as a massstorage device for storing and retrieving data. In some examples, server1140 assigns a drive letter to VLUN 1122. Server 1140 can issue storagetransactions, such as read and write transactions, over PCIe link 1162.These storage transactions can include one or more NVMe frames 1141 thatindicate an address for data to be read or written (and may include data1142 for write transactions). The address can include a logical blockaddress or object identifier, among other addresses or data identifiers.Processing system 1110 receives these NVMe frames over PCIe link 1162and handles the associated storage transactions. In a write example,data 1142 is stored in storage space 1101, specifically in logicalstorage unit 1103. Data 1142 can be striped across many storage drivesor parallelized for storage among many storage drives that comprisestorage space 1101.

In a third example, server 1150 is coupled over PCIe link 1163 usingPCIe signaling with processing system 1110 over PCIe switch 1113.External connectors and a cable carrying PCIe signaling, such as a PCIedirect connect cable can be employed. Logical storage unit 1104 ispresented as NVMe VLUN 1123 to server 1150 over PCIe link 1163. Server1150 can mount this NVMe VLUN 1123 as a logical storage unit and usethis storage space as a mass storage device for storing and retrievingdata. In some examples, server 1150 assigns a drive letter to VLUN 1123.Server 1150 can issue storage transactions, such as read and writetransactions, over PCIe link 1163. These storage transactions caninclude one or more NVMe frames 1151 that indicate an address for datato be read or written (and may include data 1152 for writetransactions). The address can include a logical block address or objectidentifier, among other addresses or data identifiers. Processing system1110 receives these NVMe frames over PCIe link 1163 and handles theassociated storage transactions. In a write example, data 1152 is storedin storage space 1101, specifically in logical storage unit 1104. Data1152 can be striped across many storage drives or parallelized forstorage among many storage drives that comprise storage space 1101.

FIG. 12 is a system diagram illustrating storage system 1200. Storagesystem 1200 includes storage space 1201 which comprises a plurality ofstorage drives contained on storage sleds. Storage space 1201 can be anexample of a storage space defined over one or more of storage sleds 110of FIG. 1, although other configurations are possible. Storage sleds 110of FIG. 1 can include more than one storage space. In this example,however, storage space 1201 is discussed. Storage space 1201 includesone or more logical storage units, such as logical storage unit 1202.These logical units can define a storage unit such as a logical unitnumber (LUN), VLUN, logical disk, among other logical storagearrangements. In this example, however, logical storage unit 1202represents an iSCSI LUN.

Some network storage protocols, such as iSCSI, allow for grouping ofdestination addresses into a logical unit. For example, iSCSI allows forportal groups which can associate more than one IP address with a singletarget. The target is a particular processor or particular networkinterface, even if that interface or processor has more than one networkaddress associated therewith. However, FIG. 12 illustrates forming alogical grouping of destination addresses for a network storage protocolusing a plurality of processors or processing nodes that share a commonstorage space spread over a plurality of storage devices. In FIG. 12,many processors can be included in a portal group, with each processorconfigured to receive network storage traffic for any of the otherprocessors and route the traffic accordingly to the correct processor,even though a single iSCSI portal group is presented to an iSCSIinitiator. In specific examples, a PCIe fabric is employed between manyprocessors, network interfaces, and storage drives. These concepts canapply to network storage protocols other than iSCSI.

Turning now to the elements of FIG. 12, storage system 1200 alsoincludes a plurality of processing systems, namely processing systems1210, 1220, 1230, 1240, and 1250. Each of the processing systems iscoupled to an associated network interface, namely network interfaces1212, 1222, 1232, 1242, and 1252. Each network interface can be coupledover an associated PCIe bus or PCIe switch with the processing system,which can include communicating over PCIe switches 1260-1263. PCIeswitches 1260-1263 comprise PCIe crosspoint switches that allowcommunication among the various processing systems. PCIe switches1260-1263 comprise a front end data plane PCIe fabric. Each processingsystem can also communicate with storage drives that comprise storagespace 1201 over ones of PCIe switches 1265-1269. PCIe switches 1265-1269comprise a back end data plane PCIe fabric. As mentioned herein, inalternative examples a unified PCIe fabric can be employed which sharesthe same PCIe switches among the front end and back end data planes.

Storage space 1201 is comprised of a one or more storage drives and eachprocessing system manages storage transactions for a subset of theplurality of storage drives. However, any processing system in storagesystem 1200 can receive a storage transaction from an external systemover their associated network interface. Although each distinct networkinterface 1212, 1222, 1232, 1242, and 1252 has a different IP addressassociated therewith, all of these distinct network interfaces aregrouped together into an iSCSI portal group. This iSCSI portal group isindicated in each processing system by IQN_(—)7. IQN_(—)7 is merely anexample portal group designation, and other portal group designationscan be employed. An iSCSI portal group can be used to associate morethan one IP address with a single target-initiator relationship usingthe iSCSI protocol. This single target can be used to receive packetsfor storage transactions over any network interface 1212, 1222, 1232,1242, and 1252. However, in this example, each network interface isassociated with a different processing system.

To allow different processing systems to receive packets for iSCSIstorage transactions but still have a shared storage space, namely LUN1202, the following process can be employed. First, a plurality ofprocessing systems, namely processing systems 1210, 1220, 1230, 1240,and 1250, are grouped into an iSCSI portal group, IQN_(—)7, whichidentifies LUN 1202 as reachable over any of IP addresses 1270-1274.Each of IP addresses 1270-1274 are associated with a different networkinterface in system 1200. iSCSI initiator 1205 can reach LUN 1202 overany of IP addresses 1270-1274 using the portal group IQN_(—)7. Eachprocessing system includes one or more central processing units (CPUs),and is coupled over a front end data plane PCIe fabric comprising PCIeswitches 1260-1263 and any associated PCIe links. If an iSCSI storagetransaction is received over any network interface then the associatedprocessing system can route the storage transaction for appropriatehandling by another processing system using the front end data planePCIe fabric. The appropriate processing system which ultimately receivesthe storage transaction then initiates a storage operation(s) withassociated storage drives that comprise storage space 1201 over anassociated back end data plane PCIe switch.

Advantageously, multiple independent processing systems (or processingnodes) can be employed within an iSCSI portal group. Any of theseindependent processing systems can receive an iSCSI storage transactionthat the receiving processing system can handle if directed towardstorage drives managed by the receiving processing system. Additionally,any of these independent processing systems can further transfer theiSCSI storage transaction to another processing system if the receivingprocessing system does not manage storage drives associated with thestorage transaction. The storage transaction typically indicates astorage address, such as a write or read address, and this address canbe used by the receiving processing system to determine if it shouldhandle the storage transaction or if it should transfer the storagetransaction to another processing system over the front end data planePCIe fabric. A PCIe address space is employed to segregate storagewithin LUN 1202 among each processing system, and this PCIe addressspace is also employed to transfer the storage transactions among thevarious processing systems. Each processing system will have anassociated address range of the PCIe address space to which it respondsduring transfer of storage transactions between processing systems.Thus, a single shared PCIe address space is employed over a PCIe fabriccomprising PCIe switches 1260-1263 and associated PCIe links.

In a specific example, packets 1203 that include write data 1204 aretransferred by iSCSI initiator 1205 for storage in LUN 1202. Thesepackets 1203 are addressed to any of IP addresses 1270-1274 and directedto the portal group indicated by IQN_(—)7. Whichever processing systemof storage system 1200 receives packets 1203 then determines whichprocessing system should actually manage the storage of write data 1204into LUN 1202. The processing system that receives packets 1203identifies a corresponding write address for write data 1204. This writeaddress can be used to select which processing system should handle thestorage transaction. Individual storage drives that comprise storagespace 1201 and LUN 1202 are each enumerated and managed by specificprocessing systems. The PCIe address space can be correlated to anaddress space of LUN 1202 which can indicate which processing systemshall handle the storage transaction. In a first example, the processingsystem that receives packets 1203 handles the write transaction becauseit identifies that it manages the storage drives associated with writedata 1204. In this first example, the receiving processing system theninitiates one or more writes indicated by packets 1203 to write data1204 into LUN 1202. These one or more writes are performed over a PCIeswitch and associated PCIe links included in the back end data plane ofsystem 1200. In a second example, the processing system that receivespackets 1203 does not handle the write transaction because it identifiesthat it does not manage the storage drives associated with write data1204. The receiving processing system then identifies the processingsystem that should handle the write transaction and transfers packets1203 over the front end data plane using associated PCIe switches andPCIe links of the front end data plane. Once the proper processingsystem receives packets 1203, then that processing system initiates oneor more writes indicated by packets 1203 to write data 1204 into LUN1202. These one or more writes are performed over a PCIe switch andassociated PCIe links included in the back end data plane of system1200.

FIG. 13 is a system diagram illustrating storage system 1300. FIG. 13includes elements indicated in FIGS. 6, 7, and 8, as well as additionalelements. FIG. 13 illustrates control plane cross connection betweenprocessing modules 630 and 730 over any of I/O module 620 and 625. Asdiscussed above, a particular processor can manage a particular I/O sledor set of SSD drives. However, data packets related to storagetransactions for a particular SSD drive can be received over any ofinterface modules 640, 641, 740, and 741. This configuration of multipleinterface modules each communicating over various TCP/IP links allowsfor load balancing, highly parallelized receipt of transactions fromexternal systems, and reception of traffic for any SSD drive over anynetwork interface included in system 1300. A particular processortypically manages traffic for a particular SSD on a PCIe back end dataplane, such as for read and write transactions to a particular storagesled, but storage transactions can be received over a network interfaceby any processor. The operations described in FIG. 13 are furtherdetailed in FIGS. 14A and 14B, which will be discussed below.

Storage system 1300 includes storage sleds 610 and 615, I/O modules 620and 625, processing module 630, interface modules 640-641, storage sleds710 and 715, processing module 730, and interface modules 740-741.Storage sled 610 includes SSD drives 611 and PCIe switch 612. Storagesled 615 includes SSD drives 616 and PCIe switch 617. Storage sled 710includes SSD drives 711 and PCIe switch 712. Storage sled 715 includesSSD drives 716 and PCIe switch 717. Although four storage sleds areincluded in FIG. 13, it should be understood that any number of storagesleds can be included. Also, although four SSD drives are included ineach storage sled, it should be understood that any number of SSD drivescan be included.

I/O module 620 includes PCIe switches 621-622, and I/O module 625includes PCIe switches 626-627. Although two PCIe switches are shown foreach I/O module in FIG. 13, it should be understood that any number canbe included. Processing module 630 includes processors 631-632 and PCIeswitches 633-634, and further elements can be included as discussedabove. Processing module 730 includes processors 731-732 and PCIeswitches 733-734, and further elements can be included as discussedabove. Processor 632 communicates over Ethernet link 840 with Ethernetswitch 833 and over Ethernet link 842 with Ethernet switch 832.Processor 732 communicates over Ethernet link 841 with Ethernet switch833 and over Ethernet link 844 with Ethernet switch 832. Processor 631and processor 731 also communicate over Ethernet switches 832-833,although associated links are omitted in FIG. 13 for clarity.

Interface module 640 communicates over at least two interfaces, namely anetwork interface 642 for TCP/IP or iSCSI traffic, and a processingmodule PCIe interface 656. Interface module 641 communicates over atleast two interfaces, namely a network interface 645 for TCP/IP or iSCSItraffic, and a processing module PCIe interface 657. Interface module740 communicates over at least two interfaces, namely a networkinterface 742 for TCP/IP or iSCSI traffic, and a processing module PCIeinterface 756. Interface module 741 communicates over at least twointerfaces, namely a network interface 745 for TCP/IP or iSCSI traffic,and a processing module PCIe interface 757.

Processing module 630 communicates with any of storage sleds 610, 615,710, and 715 over PCIe links 654 and 655 which are switched throughassociated PCIe switches on associated I/O modules. Processing module730 communicates with any of storage sleds 610, 615, 710, and 715 overPCIe links 754 and 755 which are switched through associated PCIeswitches on associated I/O modules. Storage sled 610 communicates withdifferent I/O modules over respective PCIe links 650-651. Storage sled615 communicates with different I/O modules over respective PCIe links652-653. Storage sled 710 communicates with different I/O modules overrespective PCIe links 750-751. Storage sled 715 communicates withdifferent I/O modules over respective PCIe links 752-753. The elementsof FIG. 13 form a storage system with multiple redundant PCIe linksbetween processing modules and storage sleds by way of I/O modules, suchas those discussed herein.

The storage systems described herein include a switch fabric which iscomprised of a plurality of PCIe switch elements along with associatedPCIe links. This switch fabric can include any of the PCIe switchesdiscussed herein, and can be partitioned into a back end data plane, afront end data plane, and a control plane. The control plane is employedto transfer management information, inter-processor controlcommunications, among other control information. The back end data planeis employed to transfer data to and from storage sleds. The front enddata plane is employed to transfer data to and from external interfacesand for transferring user data between processing modules. When multipleprocessors are employed, the switch fabric is preconfigured with afailover path. If the communications goes down for a particularprocessor, the hardware automatically switches over to anotherprocessor. Additionally, there is a management processor on each IOmodule that can be instructed to dynamically change the PCIe paths asnecessary. The same type of mechanisms that are employed for failoverare used to form a redundant path for each of the components the fabric.

In addition to the paths the hardware provides, there are alsosoftware-based additional layers of redundancy via frontend and backendrouting. Processors within a storage system are aware of the currentstate of monitored processors and can determine additional routes tostorage sleds as required. Hardware provides minimal latency reroutesfor failover and redundancy. While software increase the levels ofredundant paths that can be changed too. (i.e. when a hardware pathfails software can set up the next path of redundancy for anotherfailure event). There is a table included in each processor that tellsit which SSD it owns normally and which it supports when there is afailover situation. In the event of a failover an interrupt is generatedon the PCIe bus as it switches over. A discovery process of drive(s)assigned to the failed processor is initiated. This is followed by theprocessor populating discovered SSD(s) that match the failover tableinto its active table.

In operation, a specific processor of a processing module will beconfigured to manage or “own” specific SSD drives on various storagesleds. For example, processor 631 can be configured to manage SSD drives611 of storage sled 610 while processor 632 can be configured to manageSSD drives 616 of storage sled 615. In another example, processor 731can be configured to manage SSD drives 711 of storage sled 710 whileprocessor 732 can be configured to manage SSD drives 716 of storage sled715. Other combinations can be used, including a subset of SSD drives ofa particular storage sled managed by a processor and the remaining SSDdrives of that particular storage sled managed by another processor.Managing an SSD drive comprises a processor initializing the SSD driveand handling read, write, or other file or object transactions for theSSD drive. Each processor can reference a data structure stored on anon-transitory computer readable medium to determine which SSD drivesare to be managed.

However, once a particular processor has been configured to manage aparticular SSD drive, that processor might experience a failure orinterruption in normal operation. Responsive to the failure of aprocessor, such as processor 631, another processor can be configured totake over management of the SSD drives previously managed by the failedprocessor 631. To take over management of the SSD drives previouslymanaged by the failed processor 631, the other processor can initializethe SSD drives previously managed by the failed processor 631 and beginmanaging file operations, such as read and write operations, for thoseSSD drives. In another example, responsive to the failure of aprocessor, such as processor 631, another processor 732 on anotherprocessing module can be configured to take over management of the SSDdrives previously managed by the failed processor 631. Processor 732 caninitialize the SSD drives previously managed by the failed processor 631and begin managing file operations, such as read and write operations,for those SSD drives. A processor can include a failover table whichindicates SSD drives for which the processor manages in the event of afailure of another processor. A failover table can also indicate aprocessor or processors that a processor monitors to determine if anyfailures have occurred.

For example, processor 632 can monitor another processor, such asprocessor 631, using different methods. In a first example, processor632 sends periodic messages to processor 631 and awaits responsemessages from processor 631. If the response messages fail to comewithin a predetermined amount of time, then processor 632 can determinethat processor 631 has failed. The periodic messages can include probepackets sent through a network stack or stacks between processors, suchas over a network interface that processor 631 and 632 can communicate.In another example, a set of registers or bits are maintained forprocessor 631 in processor 632 that are set or reset by processor 631.If a register or bit associated with processor 631 has not been properlyset or reset within a predetermined amount of time, then processor 632can determine that processor 631 has failed. Although two processors areshown in FIG. 13 as monitoring each other, in other examples a greaternumber of processors can be included. This greater number of processorscan each be configured be monitored for failure by all other processors,by only one other processor (dual mode), or by two other processors(tri-mode). Any combination of monitored and monitoring processors canbe employed.

Once a processor has been determine to have failed, the monitoringprocessor, such as processor 632, can be configured to initiate arestart, reboot, or power cycle of the failed processor. If the failedprocessor comes back online and functioning after the restart, reboot,or power cycle, then those SSD drives that were transitioned to themonitoring processor can be re-managed by the previously failedprocessor.

To handle incoming data packets originally intended for a failedprocessor, another processor can inherit network connection states fromthe failed processor, such as TCP/IP connections, and subsequentlyhandle network traffic originally handled by the failed processor. Onthe front end (TCP/IP stack on each operating system (OS)), connectedexternal iSCSI clients seamlessly migrate to the new owning node. Thismigration does not require a disconnect and reconnect. From the frontend point of view, each iSCSI connection when the client initiates theTCP connection (3-way handshake), OS enhancements to the TCP stackinclude propagating the connection state (i.e. the source/destination IPaddress and the source/destination port ID) to all the processingmodules in the cluster. The processing modules in the cluster then spooftheir local TCP/IP stacks by creating connected socket on their localstacks as well. These sockets are created and ready but never used untilfailover. When the node failover event is initiated in addition to allthe steps described earlier, the node that is assuming ownership of theTCP connections enables the pre-connected client connections that werepreviously connected to the crashed node. Further details on thisprocess are described below in FIGS. 3-5.

In other examples, an operating system backend software invalidateslayer-2 (Ethernet) routing table entries pointing to the crashed nodeand generates new layer-2 (Ethernet) routing tables entries that pointto the new owning peer as the destination for IOs that access theSSDs/LBAs contained on the transferred SSDs. These tables are propagatedto all the nodes in the cluster. Incoming iSCSI IOs now are routedcorrectly at layer-2 to the new owning node.

As a specific example, FIG. 13 illustrates one example failover process.In FIG. 13, a network connection is established between processor 632and an external iSCSI node, such as an end user device or other computersystem. The network connection is typically defined by at least anetwork address and a network port for each endpoint, such as an IPaddress and port a network interface card of interface module 641 andthe external iSCSI node. The network connection can be referred to as a“socket” in some examples. Once this network connection is established,processor 632 places information regarding the network connection in adata structure, such as connection table 1380. Connection table 1380 canlist several network connections that processor 632 is handling. Thenetwork connection can be used to handle storage transactions, such asstorage transaction 1394, between the external iSCSI node and a storagedrive, such as found on storage sled 616.

Responsive to establishing the network connection, processor 632 alsopropagates information about the network connection to furtherprocessing modules or processors in system 1300. For example, connectionstate transfer 1393 indicates a transfer of connection information fromconnection table 1380 to processor 732 over Ethernet control plane links840-844 in FIG. 13. Processor 732 can store this connection informationin connection table 1381, in addition to other connection informationreceived from other processors or processing modules.

If processor 632 operates normally, then storage transactions for theassociated network connection are handled by processor 632. However, ifprocessor 632 becomes unresponsive, such as due to a failure, hang, orother problem, then another processor can inherit the network connectionoriginally established by processor 632. For example, processor 732 canbe configured to monitor processor 632, as described above, anddetermine when processor 632 has failed. Responsive to the failure ofprocessor 632, processor 732 can inherit any TCP/IP connections withexternal iSCSI nodes and subsequently handle any storage transactionsinitiated by those external iSCSI nodes. These storage transactions canbe for storage drives, such as found on storage sled 615, that wereoriginally managed by processor 632. Thus, processor 732 can assumemanagement of associated storage drives on storage sled 615 and handlesubsequent storage transactions for those storage drives, in addition toany storage drives that processor 732 might already be managing.

As further examples of TCP/IP connections state failover from oneprocessor to another, FIGS. 14A, 14B, and 15 are presented. FIGS. 14Aand 14B are system diagrams illustrating system 1400, which can includecommon elements as found in FIG. 13, although variations are possible.FIG. 15 is a flow diagram illustrating a method of operating system1400. The operations of FIG. 15 are referenced below parenthetically.

System 1400 includes processing system 1410, external system 1490,processing system 1420, and Ethernet switch 1430. Processing system 1410includes network interface 1411 and connection state information 1412.Processing system 1420 includes network interface 1421 and connectionstate information 1422. Processing system 1410 and 1420 can includefurther elements, such as the processors, storage systems, software, andother elements described herein for processors and processing modules.Network interfaces 1411 and 1421 include network interface equipment andtransceivers, along with any associated software drives and protocolstacks, such as TCP/IP stacks.

In operation, processing system 1410 establishes a network connection1470 with external system 1490 over a packet network link (1501). Thenetwork connection can comprise a TCP/IP connection, which can transportstorage transactions and information related to further protocols, suchas iSCSI. Processing system 1410 establishes the network connectionusing at least network interface 1411, which in this example has atleast one associated IP address and port for the network connection,namely IP address 1460 and port 1461. External system 1490 has at leastone associated IP address and port for the network connection, namely IPAddress 1450 and port 1451. Processing system 1410 records networkconnection information for the network connection established betweenprocessing system 1410 and external system 1490, which indicates acombination of the IP addresses and ports. Processing system 1410 canstore the network connection information in a data structure, such asconnection state information 1412. This network connection informationis indicated in connection state information table 1480. Table 1480lists network connection 1470 as comprising source IP address and port1450, 1451, and destination IP address and port 1460, 1461. Differentcombinations of this information can be used to represent the networkconnection, along with additional or different information.

Processing system 1410 also propagates or distributes informationdescribing network connection 1470, such as that found in table 1480, toother processing systems in system 300 (1502). These other processingsystems are referred to as monitoring peers, and can include one or moreprocessors. Typically, 2-3 processors are employed as peers to any otherprocessor, although variations are possible, including up to ‘n’processors. Processing system 1410 propagates this network connectioninformation over a control plane, which comprises at least Ethernetswitch 1430 along with any associated Ethernet links. Although onlyprocessing system 1420 is shown as an additional processing system, itshould be understood that many processing modules or processing systemscan be included, such as found in FIG. 13. Connection state information1480 is received by processing system 1420 and stored in a datastructure, such as connection state information 1422. Further processingsystems can receive and store this connection state information. In thisexample, processing system 1420 or network interface 1421 are associatedwith at least network address 1465 and port 1466.

Peer processing system 1420 establishes an inactive network connectionwhich is identical to the connection 1470 established by processingsystem 1410, namely the connection or socket between processing system1410 and external system 1490 (1511). However, processing system 1420maintains this network connection as an ‘idle’ or ‘inactive’ connectionand does not direct any network traffic over this connection. Furtherprocessing systems can also establish these identical sockets orconnections. Only processing system 1410 directs network traffic overthis connection, at least until processing system 1410 fails.

External system 1490 can initiate storage transactions, such as readsand writes, for data stored in associate storage drives. Packets 1401represent these storage transactions, and can indicate storageaddresses, write data, and other information. Processing system 1410manages one or more storage drives and transfers or receives data 1402to/from these storage drives, such as over PCIe interfaces to/fromstorage sleds as found in FIG. 13.

In FIGS. 14A and 14B, processing system 1420 monitors processing system1410 (1503). In this example, processing system 1420 is a monitoringpeer of processing system 1410. Processing system 1420 can detect whenprocessing system 1410 has hung or otherwise failed (1504), andprocessing system 1420 can inherit network connection 1470 originallyestablished and managed by processing system 1410. First, processingsystem 1420 disables any receipt of packets for connection 1470established by processing system 1410 (1505). Specifically, processingsystem 1420 does not pass any packets for connection 1470 up a networkstack. These packets can be dropped and not responded to, and anypackets received by an associated IP stack of processing system 1420 forconnection 1470 are dropped.

A MAC address associated with processing system 1420 is established tobe associated with the inherited connection (1506). An ARP message isbroadcast over the subnet of processing system 1420 to associate IPaddress 1460 of the crashed node (processing system 1410) with a MACaddress of processing system 1420. In network systems employing the IPprotocol, an Address Resolution Protocol (ARP) messages can be used toresolve an IP address into an associated Media Access Control (MAC)address. Network routers and other network nodes can employ an ARP tablethat lists relationships between IP addresses and MAC addresses. The IPaddresses typically represent layer-3 addresses, while the MAC addressestypically represent layer-2 addresses, such as Ethernet.

A probe packet is sent by processing system 1420 for delivery toexternal system 1490 to establish correct acknowledge and sequence(ack/seq) counts for the packets associated with the inheritedconnection (1507), and this probe packet process is described below inmore detail. The probe packet can include zeroed out counts inacknowledge and sequence portions of the TCP header. TCP operationincludes “stateful” operation, where a current packet state ismaintained by the endpoints of a TCP network connection. The TCP stateincludes address and port information for each endpoint as well aspacket counters for traffic exchanged between the endpoints to guaranteemissing packets are accounted for and resent. In the examples discussedherein, TCP state is maintained even when one of the endpoints fails,namely a processor or network interface handling the connection. Uponinitiation of a stateful TCP network connection, various packet countersare set to an initial value, such a ‘0’ and each subsequent packetexchanged increments the counters by one. These counters are embedded inheaders of the associated TCP packets, and separate counters might beincluded for a “packet” number and an “acknowledgement received” number.If an endpoint notices that the packet counter is out of sequence withthe acknowledgement sequence, such as when a packet is dropped ormissing, then a packet resend process is initiated which ensures thatmissing packets are properly delivered between the endpoints.

Advantageously, instead a peer processing system keeping track of thestate ack/seq counts for connection 1470 that can change quickly duringa communication session, the probe packet allows for derivation of theack/seq state information from external system 1490 at the time of thefailover from processing system 1410 to processing system 1420. Theack/seq state information need not be stored or maintained by a peerprocessing system during the time when connection 1470 is active onprocessing system 1410.

When a new processing system, such as processing system 1420, attemptsto take over or inherit a network connection from a failed processingsystem, such as processing system 1410, processing system 1420 might nothave all the needed information to resume communications. For example,although processing system 1420 might have recorded the networkconnection state information 1480, which includes IP address and portnumbers for connection 1470, processing system 1420 does not know acurrent packet count or packet counter information for the networkconnection between processing system 1410 and external system 1490.Responsive to processing system 1410 failing, processing system 1420 cansend an acknowledgement (ACK) packet for delivery to external system1490 with a nonsense counter value, such as ‘0’. This ACK packet can bereferred to as a ‘probe’ packet.

Processing system 1420, i.e. the monitoring peer, enables receipt andprocessing of packets received for connection 1470 (1508). Externalsystem 1490 receives the probe packet and detects that the associatedpacket counter is out of sequence or has incorrect numbers, andresponsively transfers the connection state, including current values ofthe packet and acknowledgement counters, in a response packet. Externalsystem 1490 responds to the probe packet with correct acknowledge andsequence counts (1509). Responsively, processing system 1420 splices thereceived ack/seq counts into status information for connection 1470 andused for subsequent communications over connection 1470 with externalsystem 1490.

Once processing system 1420 has inherited the network connectionoriginally established between processing system 1410 and externalsystem 1490, then processing system 1420 can begin handling storagetransactions for that network connection without interrupting, breaking,ending, or restarting that network connection (1510). Specifically,processing system 1420 can continue to receive packets 1401 for storagetransactions originally handled by processing system 1410, such as forstorage transactions related to data 1402 and storage drives originallymanaged by processing system 1410. Processing system 1420 can assume IPaddress 1460 and port 1461 from failed processing system 1410.Processing system 1420 can resume transfer of packets 1401 with externalsystem 1490 as if processing system 1420 was processing system 1410, andseamlessly inherits the network connection of failed processing system1410. All subsequent iSCSI traffic for connection 1470 now flows to theprocessing node 1420.

During the failure of processing system 1410, peer processing system1420 can attempt a reboot of processing system 1410 (1512) to attempt torecover operation of processing system 1410. If processing system 1410recovers, then processing system 1420 can either keep the currentconnection that was failed over to processing system 1420 fromprocessing system 1410, or processing system 1420 can transfer thatconnection back to processing system 1410.

FIG. 16 is a system diagram illustrating storage system 1600. Storagesystem 1600 illustrates several storage enclosures stacked vertically,such as in a rackmount enclosure system. In some examples, the storageenclosures comprise 3U sized rackmount enclosures, although other sizescan be employed. Any number of storage enclosures can be included, andfurther storage enclosures can be included in other non-verticallystacked configurations or in a plurality of rack mount enclosures.

System 1600 includes four example storage enclosures, namely 1610, 1620,1630, and 1640. Each storage enclosure comprises a storage system, suchas described in FIG. 1 and FIG. 2 above. Two I/O modules are shown perstorage enclosure, which can each comprise I/O modules as found in FIGS.1-3. Each I/O module includes at least four external connectors that canbe used to interconnect I/O modules of other storage enclosures. Forexample, storage enclosure 1610 includes two I/O modules 1611. These I/Omodules can be user-insertable into the respective storage enclosure,such as by sliding in and out of an associated midplane connector andmounting system.

The external cluster interconnect for PCIe front end data planecommunications is provided by a plurality of mini serial attached SCSI(SAS) connectors 1612, such as the four mini-SAS connectors 1612 shownper I/O module. These mini-SAS connectors can comprise mini-SAS jacks.Cabling comprises SAS cabling 1613 which can include associatedshielding, wiring, sheathing, and termination connectors. The externalcluster interconnect for Ethernet control plane communications isprovided by a plurality of Ethernet connectors 1614 and associatedEthernet cabling 1615. These Ethernet cables 1615 can be directlyconnected between storage enclosures or can be routed to a centralEthernet switch, not shown in FIG. 16.

In this example, each front end data plane cluster interconnectconnector and cable do not carry SAS signaling. Instead, thesecomponents carry PCIe signaling between crosspoint switches of eachassociated I/O module. Thus, each storage enclosure intercommunicatesover PCIe signaling carried over mini-SAS connectors and mini-SAScabling. In this example, ×16 PCIe signaling is employed, although otherPCIe widths can be employed. Since eight ×16 PCIe signaling pathways areemployed per storage enclosure, then 128 lanes of PCIe signaling areprovided for each storage enclosure. This can lead to at least 32petabytes of addressable storage space with 32 terabytes per second ofthroughput. Additionally, no centralized switching elements, such asEthernet switches, are required in this front end data plane PCIecluster interconnect, and traffic for each storage enclosure forms asystem-wide “fabric” over which the various elements of the cluster cancommunicate.

Any processor in the cluster can receive packets over an associatedexternal interface, such as TCP/IP, for storage transactions thatinvolve any storage drive in any storage enclosure. A receivingprocessor can route these packets over the PCIe fabric to a managingprocessor that manages a specific set of storage drives. The entirecluster comprising system 1600 can be presented as a single storageentity, such as logical storage unit, and by using the cluster-wide PCIefabric any processor has memory mapped visibility to any storage drivein the entire cluster. To route storage transactions to any otherprocessor, a receiving processor can perform a PCIe transaction to amemory location that is associated with managing processor, even if themanaging processor is external to the storage enclosure of the receivingprocessor and is associated with different I/O modules of that otherstorage enclosure. Traffic is automatically routed over the PCIe fabricby the associated crosspoint switches of the associated I/O modules andany external PCIe links to be handled by the appropriate processor. Eachprocessor of the cluster includes a routing table of that indicateswhich address is associated with a specific storage location of theentire logical storage unit of the cluster.

For example, a first processor of storage enclosure 1610 receives awrite transaction over an associated network interface from an end userdevice. This first processor checks a routing table associated with thefirst processor to identify a PCIe address associated with the writetransaction. The write transaction can indicate a storage address forthe write transaction, which can include a storage address of a logicalstorage unit presented by system 1600 to the end user, such as an iSCSIstorage device presented over one or more network interfaces. The firstprocessor can then transfer the storage transaction to the PCIe addressassociated with the write transaction. The PCIe fabric in this exampleincludes multiple storage enclosures, and thus the PCIe address can beassociated with any processor in system 1600, such as a second processorincluded in storage enclosure 1620. I/O modules of enclosure 1610 andenclosure 1620 handle the PCIe transaction to ensure that the secondprocessor receives the storage transaction. Once the second processorreceives the storage transaction, the second processor determines thatit handles storage transactions for the particular storage drivesindicated by the storage transaction and writes associated data tostorage drives to complete the storage transaction. In this manner, anystorage transaction can be received by any processor in the clusterformed by system 1600 and the receiving processor can deliver thatstorage transaction to another processor across the cluster that managesthe storage drives associated with the storage transaction. The mini-SASconnectors and cabling are employed to carry PCIe signaling between I/Omodules of the various storage enclosures.

To further illustrate the back end data plane of the storage systemsdiscussed herein, FIG. 17 is presented. FIG. 17 is a system diagramillustrating storage system 1700. Storage system 1700 includes manyelements of a storage system as described in the many examples herein.However, storage system 1700 focuses on the management andidentification of the various storage sleds in a processing system, suchas storage sleds 1710 and 1715, among others. Management processor 1725is included in I/O module 1720. Management processor 1725 communicateswith storage sleds over associated USB interfaces, namely USB interfaces1751 and 1752. Management processor 1725 communicates with the variousprocessing systems or processing nodes of a storage system over Ethernetbus 1750, which can include network switch 1726. This USB/Ethernetcommunication system allows for initialization, control, and statusreporting of the various storage sleds to any processor or processingnode of a storage system. The USB/I2C/Ethernet interfaces described inFIG. 17 can comprise the various “sideband” interfaces discussed hereinfor management and monitoring of the various storage sleds in a storagesystem by the main processors or processing nodes of a storage system.

For example, any of the processors or processing nodes of FIG. 17 canreceive status information above any of the storage sleds over thisEthernet bus 1750. Management processor 1725 monitors and collectsstatus information, operating statistics, among other information foreach storage sled over USB interfaces. Each storage sled includes atleast one processor (1713, 1718) which can be accessed by managementprocessor 1725 over the associated USB interface.

Each processor on a storage sled can interface with an associated PCIeswitch to receive operating statistics, status information, or operatingconditions, among other information. This information collected on eachstorage sled can be further accumulated in one or more I/O modules by anassociated processor on the I/O module, such as management processor1725. This information can be stored in computer-readable memoryelements associated with management processor 1725. The informationabout each storage sled can be accumulated by one or more I/O modulesfor redundancy, failover protection, and load balancing in a storagesystem. The processor on the storage sleds can interface with theassociated PCIe switch of the storage sled via I2C interfaces, amongother register-based interfaces. Each PCIe switch can include externallyaccessible information and registers which can be accessed and modifiedover an associated interface, such as interfaces 1714 and 1719 in FIG.17. Interfaces 1714 and 1719 can comprise I2C interfaces, among otherinterfaces.

As a storage system operates, usage statistics among other informationis monitored by a storage sled processor and transferred or requested bya further processor on any I/O module. This usage information can befurther transferred or requested by main processing modules orprocessors over Ethernet bus 1750 which can be used to alter or modifyoperation of the storage system or for usage statistics monitoring andreporting to other external systems.

In addition to monitoring of status information and operationalstatistics of a storage system, the sideband interfaces described inFIG. 17 can be used for initialization of the various storage sleds of astorage system, among other initialization features. Specifically, uponpower up of a storage system, such as storage system 1700, managementprocessor 1725 can assign unique identifiers to each storage sled overthe associated USB interfaces. These unique identifiers can be used toidentify each storage sled for retrieval of status information andoperational statistics by main processors. These identifiers can bereported to the main processors over Ethernet bus 1750. The identifierscan comprise 16-bit identifiers which are used in combination oralternatively to any 64-bit PCIe identifiers for the storage sleds.These identifiers can correspond to a physical location of a storagesled, such as physical slots on a backplane or midplane, among otherphysical locations within a storage unit.

Now, the initialization process for the back end data plane will bediscussed. Each processor, such as the various processing modulesdiscussed herein, only manages a certain number of storage drives, suchas found on a storage sled. The PCIe switches employed herein canestablish segregated connectivity between a particular processor and astorage sled. For example, processing module 1701 of FIG. 17 can beinitially configured by PCIe switches 1721 to communicate with storagesled 1710 over PCIe links 1740 and 1742. Likewise, processing module1702 of FIG. 17 can be initially configured by PCIe switches 1721 tocommunicate with storage sled 1715 over PCIe links 1741 and 1743. Ifprocessing module 1702 fails or becomes unresponsive, then PCIe switches1721 can be configured to fail over connectivity for storage sled 1715to processing module 1701, or another processing module. This can beseen in FIG. 17 as initial configuration 1760 and failover configuration1761 for processing module 1701.

PCIe switches 1721 can be managed by management processor 1725, such asover a management interface which can comprise I2C, an EEPROM interface,serial interface, parallel bus, or other interfaces. Upon power up of astorage system, such as system 1700, management processor 1725 canensure that PCIe switches 1721 do not provide connectivity between anyprocessing modules and any storage sleds. Management processor 1725 canselectively power up individual PCIe switches (such as when many areemployed as found in FIG. 1) and scan for attached storage sleds.Alternatively, management processor 1725 can scan for attached storagesleds by powering on selective identification circuitry on each storagesled. In yet other examples, each storage sled contains a pull-up pinwhich indicates when an associated storage sled has been inserted into amidplane connector. Management processor 1725 can detect which storagesleds are present and establish connectivity between various processingmodules and various storage sleds using PCIe switches 1721.

In some examples, a failover event is accompanied by a reprogramming ofregisters or configuration parameters within PCIe switches 1721, andmanagement processor 1725 can be configured to perform thisreprogramming. Management processor 1725 can be also instructed by anyprocessing module, such as processing modules 1701 or 1702 over Ethernetinterface 1750 to change or alter the PCIe connectivity betweenprocessing modules and storage sleds.

In another example, ones of PCIe switches 1721 can be initiallyprogrammed for a power-up or ‘default’ configuration that establishesconnectivity between various PCIe interfaces handled by each of PCIeswitches 1721. A failover configuration can also be pre-established bymanagement processor 1725. In the event of a detected failure of aprocessing module, management processor 1725 can establish the failoverconfiguration. The failover and default configurations can be maintainedin a data structure, such as a table, in a storage element associatedwith management processor 1725. The failover configuration can beestablished in PCIe switches by changing registers or configurationinformation of an associated PCIe switch over an I2C interface. A rebootof the associated PCIe switch may be required in some examples. Once thefailover connectivity is established, then the processing module whichhas been newly connected to the storage sled can perform aninitialization process with the storage drives on the storage sled, suchas by performing a PCIe enumeration process.

During a boot up process for the entire storage system, managementprocessor 1725 can be configured to detect all storage drives andstorage sleds in the storage system and report this information to eachprocessing module of the system over Ethernet interface 1750. Eachprocessing module, alone or in concert, can be configured to establishwhich storage sleds will be apportioned to which processing module.Management processor 1725 can receive instructions from one or moreprocessing modules to establish a desired connectivity between ones ofthe storage sleds and ones of the processing modules using theconfigurable PCIe interfaces discussed above.

Each storage sled, or each storage drive on the storage sleds, can beidentified by a unique identifier by management processor 1725 for usein identification of particular storage sleds by each processing module.For example, if a storage system has 0-47 storage sleds, then eachstorage sled can have a unique identifier assigned by managementprocessor 1725 during power up. Any communications between managementprocessor 1725 and the various processing modules that need to identifya storage sled can employ these unique identifiers, such as controlplane communications over Ethernet interface 1750.

FIG. 18 is a block diagram illustrating processing system 1800.Processing system 1800 illustrates an example of any of the controlprocessors, power control elements, or sled processors discussed herein,such as processing systems 131 of FIG. 1, processor 320 or powercontroller 321 of FIG. 3, processor 450 and power controller 422 of FIG.4, processor 511 or power controller 531 of FIG. 5, or processors 1713,1718, and 1725 of FIG. 17. In addition, processing system 1800 can beillustrative of any processing system, processor, or CPU on a processingmodule, such as processing system 131 of FIG. 1, among others.

Control processor 1800 includes communication interface 1801 andprocessing system 1810. Processing system 1810 includes processingcircuitry 1811, random access memory (RAM) 1812, and storage 1813,although further elements can be included. Example contents of RAM 1812are further detailed in RAM space 1820, and example contents of storage1813 are further detailed in storage system 1860.

Processing circuitry 1811 can be implemented within a single processingdevice but can also be distributed across multiple processing devices orsub-systems that cooperate in executing program instructions. Examplesof processing circuitry 1811 include general purpose central processingunits, microprocessors, application specific processors, and logicdevices, as well as any other type of processing device. In someexamples, processing circuitry 1811 includes physically distributedprocessing devices, such as cloud computing systems.

Communication interface 1801 includes one or more communication andnetwork interfaces for communicating over communication links, networks,such as packet networks, the Internet, and the like. The communicationinterfaces can include serial links, such as SPI links, I2C links, USBlinks, UART links, or one or more local or wide area networkcommunication interfaces which can communicate over Ethernet or Internetprotocol (IP) links. Communication interface 1801 can include networkinterfaces configured to communicate using one or more networkaddresses, which can be associated with different network links.Examples of communication interface 1801 include network interface cardequipment, transceivers, modems, and other communication circuitry.

RAM 1812 and storage 1813 together can comprise a non-transitory datastorage system, although variations are possible. RAM 1812 and storage1813 can each comprise any storage media readable by processingcircuitry 1811 and capable of storing software. RAM 1812 can includevolatile and nonvolatile, removable and non-removable media implementedin any method or technology for storage of information, such as computerreadable instructions, data structures, program modules, or other data.Storage 1813 can include non-volatile storage media, such as solid statestorage media, flash memory, phase change memory, or magnetic memory,including combinations thereof. RAM 1812 and storage 1813 can each beimplemented as a single storage device but can also be implementedacross multiple storage devices or sub-systems. RAM 1812 and storage1813 can each comprise additional elements, such as controllers, capableof communicating with processing circuitry 1811.

Software stored on or in RAM 1812 or storage 1813 can comprise computerprogram instructions, firmware, or some other form of machine-readableprocessing instructions having processes that when executed a processingsystem direct control processor 1800 to operate as described herein. Forexample, software can drive processor 1800 to monitor operatingstatistics and status for various storage sleds and other modules,monitor power status for the sleds and modules, and instruct powercircuitry to control flow of holdup power or operational power, amongother operations. The software can also include user softwareapplications. The software can be implemented as a single application oras multiple applications. In general, the software can, when loaded intoa processing system and executed, transform the processing system from ageneral-purpose device into a special-purpose device customized asdescribed herein.

RAM space 1820 illustrates a detailed view of an example configurationof RAM 1812. It should be understood that different configurations arepossible. RAM space 1820 includes applications 1821 and operating system(OS) 1822. Software applications 1823-1825 each comprise executableinstructions which can be executed by processor 1800 for operating aprocessor, CPU, controller, or other circuitry according to theoperations discussed herein.

Failure monitor 1823 monitors processing systems, such as processors 131on processing modules 130, PCIe switches, storage sleds, storage drives,and other components to detect and identify when any of these componentsexperiences failures or degraded performance. The failures can bedetected based on keep-alive messaging, periodic polling, discretesignal indicators, or other periodically or interrupt-based indicators.In some examples, the failures are detected using statisticalmonitoring, where performance statistics, such as throughputs, bandwidthutilization, processor utilization, memory utilization, or otherstatistics are monitored to identify when a component is experiencingfailure or degradation. Failure monitor 1823 can report failures ordegraded performance to failover module 1824.

In periodic polling examples, a first processor can be configured toperiodically transfer a polling message for delivery to a secondprocessor monitored by the first processor. This polling message can bereceived by the first processor and a response can be transferred to thesecond processor which can then determine that the first processor hasnot yet failed. Polling messages can be combined with other processes,such as monitoring traffic levels handled by a particular processor todetermine a robust determination of when a processor is unresponsive.Polling messages can be transferred over more than one communicationpathway, which can lead to a determination of when a particularcommunication pathway is experiencing failures or problems. The firstprocessor can likewise transfer polling messages for delivery to thesecond processor in a similar manner to monitor failure of the secondprocessor. More than one processor can be configured to poll more thanone other processor. For example, PCIe messages can be exchanged amongthe various processors and processing modules of FIG. 1 to allowidentification of when a processor is unresponsive and possibly failed.

Failover module 1824 can initiate failover processes for the variousprocessors that handle storage transactions, such as processors 131 onprocessing modules 130, among other components. If a processor fails,such as discussed in FIGS. 6-7, then failover module 1824 canresponsively switch management of storage drives or storage modules toanother non-failed processor to handle further data transactionsdirected to storage drives originally associated with the failed one ofthe processors. Likewise, if a PCIe switch fails, then traffic of thatPCIe switch can be re-routed to redundant PCIe switches to enablecontinued operation and storage operation handling.

Initialization module 1825 initializes storage sleds and storage drivestherein, such as by instantiating various storage drives and assigningvarious addresses and identification numbers to the various storagedrives and storage sleds. Initialization module 1825 can also initializevarious parts on the various modules discussed herein, such as PCIeswitches, network interfaces, power control circuitry, power controlprocessors, sideband interfaces, among other elements. Initializationmodule 1825 can communicate over control plane and sideband interfacesto perform the initialization.

PCIe switch module 1826 handles initialization, configuration, andmanagement of the various PCIe switches and links discussed herein. PCIeswitch module 1826 can establish non-transparent port configurations,domains for logical PCIe segregation, among other operations. PCIeswitch module 1826 can also retrieve and monitor statistical informationfor PCIe switches and links to provide these to other processors formonitoring and presentation over user interfaces to users or operators.

Power module 1827 monitors usage status or usage statistics for elementsof sleds and modules. The usage statistics include data transfer ratesof links, error rates of links, a cumulate number of errors of links,among other statistics. The usage statistics can be collected and storedby processor 1800 in a data structure, such as a database or table andstored in storage 1813, RAM 1812, or other storage elements. Powermodule 1827 monitors power inrush statistics during a power-up process,power status statistics, power active status, voltage levels, phasemeasurements, current draw, holdup circuit status or levels, sled/moduleinsertion status, thermal levels, among other statistics. Power module1827 instructs power circuitry to power up or power down an associatedsled or module using signals such as discrete signals of associatedpower circuitry. Power module 1827 can power up or power down a sled ormodule responsive to data commit status of associated storage drives orother circuitry, responsive to insertion status, or other factors.

Applications 1821 and OS 1822 can reside in RAM space 1820 duringexecution and operation of control processor 1800, and can reside instorage system 1860 during a powered-off state, among other locationsand states. Applications 1821 and OS 1822 can be loaded into RAM space1820 during a startup or boot procedure as described for computeroperating systems and applications.

Storage system 1860 illustrates a detailed view of an exampleconfiguration of storage 1813. Storage system 1860 can comprise flashmemory such as NAND flash or NOR flash memory, phase change memory,magnetic memory, among other solid state storage technologies. As shownin FIG. 18, storage system 1860 includes system software 1861. Asdescribed above, system software 1861 can be in a non-volatile storagespace for applications and OS during a powered-down state of controlprocessor 1800, among other operating software.

Control processor 1800 is generally intended to represent a computingsystem with which at least software 1861 and 1821-1822 are deployed andexecuted in order to render or otherwise implement the operationsdescribed herein. However, control processor 1800 can also represent anycomputing system on which at least software 1861 and 1821-1822 can bestaged and from where software 1861 and 1821-1822 can be distributed,transported, downloaded, or otherwise provided to yet another computingsystem for deployment and execution, or yet additional distribution.

The included descriptions and figures depict specific embodiments toteach those skilled in the art how to make and use the best mode. Forthe purpose of teaching inventive principles, some conventional aspectshave been simplified or omitted. Those skilled in the art willappreciate variations from these embodiments that fall within the scopeof the invention. Those skilled in the art will also appreciate that thefeatures described above can be combined in various ways to formmultiple embodiments. As a result, the invention is not limited to thespecific embodiments described above, but only by the claims and theirequivalents.

What is claimed is:
 1. A data storage system, comprising: a plurality ofstorage drives each comprising a Peripheral Component InterconnectExpress (PCIe) interface, and configured to store data and retrieve thedata stored on associated storage media responsive to storageoperations; one or more processing modules comprising one or moreprocessors communicatively coupled to the plurality of storage drivesover a PCIe fabric comprised of one or more PCIe switches; theprocessors configured to share a PCIe address space associated with thePCIe fabric for transfer of the storage operations to appropriate onesof the processors that manage ones of the plurality of storage drives.2. The data storage system of claim 1, comprising: a first of theprocessors configured to receive a first storage operation and identifya portion of the PCIe address space associated with the first storageoperation; when the portion of the PCIe address space is associated withthe second of the processors, the first of the processors configured totransfer the first storage operation for delivery to a second of theprocessors over the PCIe fabric for handling of the storage operationwith ones of the plurality of storage drives assigned to the second ofthe processors.
 3. The data storage system of claim 1, comprising: afront end data plane formed among at least a first one of the PCIeswitches that communicatively couples the processors to each other; anda back end data plane formed among at least a second one of the PCIeswitches that communicatively couples the processors to the plurality ofstorage drives.
 4. The data storage system of claim 1, comprising: aunified data plane formed among the one or more PCIe switches thatcommunicatively couples the processors to each other, andcommunicatively couples the processors to the plurality of storagedrives.
 5. The data storage system of claim 1, comprising: the one ormore processors communicatively coupled to network interfaces over thePCIe fabric, the network interfaces configured to receive the storageoperations over associated network links and transfer the storageoperations to the one or more processors over the PCIe fabric.
 6. Thedata storage system of claim 5, comprising: each of the networkinterfaces configured to be associated with a different InternetProtocol (IP) address and at least two of the network interfacesconfigured to be associated with a first Internet Small Computer SystemInterface (iSCSI) portal group; a first processor associated with afirst network interface and associated with the first iSCSI portal groupconfigured to identify data packets received over the first networkinterface as comprising a storage operation associated with at least oneof the plurality of storage drives managed by a second processor, andresponsively transfer the first data packets for handling by theprocessor having a second network interface associated with the firstiSCSI portal group; and the second processor configured to handle thestorage operation associated with the data packets as if the secondnetwork interface had received the data packets.
 7. The data storagesystem of claim 1, comprising: at least one of the PCIe switchesconfigured to interconnect processing modules of the data storage systemwith another data storage system over at least one external PCIe link.8. The data storage system of claim 7, comprising: the at least oneexternal PCIe links comprising PCIe signaling carried over SerialAttached Small Computer Serial Interconnect (SAS) connectors and cablingto interconnect the data storage system to the other data storagesystem.
 9. The data storage system of claim 1, comprising: at least oneof the processors configured to present an associated one or morestorage drives as at least one NVM Express (NVMe) virtual logical unitnumber (VLUN) over a PCIe interface that communicatively couples the atleast one of the processors and a computing system external to the datastorage system; the at least one of the processors configured to receiveNVMe storage transactions transferred by the computing system over thePCIe interface and handle the NVMe storage transactions for storage ofdata on the associated one or more storage drives.
 10. A method ofoperating a data storage system, the method comprising: in a pluralityof storage drives, storing data and retrieving the data stored onassociated storage media responsive to storage operations received overa Peripheral Component Interconnect Express (PCIe) interface; in one ormore processing modules comprising one or more processorscommunicatively coupled to the plurality of storage drives over a PCIefabric comprised of one or more PCIe switches, sharing a PCIe addressspace associated with the PCIe fabric for transfer of the storageoperations to appropriate ones of the processors that manage ones of theplurality of storage drives.
 11. The method of claim 10, furthercomprising: in a first of the processors, receiving a first storageoperation and identifying a portion of the PCIe address space associatedwith the first storage operation; when the portion of the PCIe addressspace is associated with the second of the processors, in the first ofthe processors, transferring the first storage operation for delivery toa second of the processors over the PCIe fabric for handling of thestorage operation with ones of the plurality of storage drives assignedto the second of the processors.
 12. The method of claim 10, wherein afront end data plane is formed among at least a first one of the PCIeswitches that communicatively couples the processors to each other; andwherein a back end data plane is formed among at least a second one ofthe PCIe switches that communicatively couples the processors to theplurality of storage drives.
 13. The method of claim 10, wherein aunified data plane is formed among the one or more PCIe switches thatcommunicatively couples the processors to each other, andcommunicatively couples the processors to the plurality of storagedrives.
 14. The method of claim 10, further comprising: communicativelycoupling the one or more processors to network interfaces over the PCIefabric; in the network interfaces, receiving the storage operations overassociated network links and transferring the storage operations to theone or more processors over the PCIe fabric.
 15. The method of claim 14,further comprising: associating each of the network interfaces with adifferent Internet Protocol (IP) address and associating at least two ofthe network interfaces with a first Internet Small Computer SystemInterface (iSCSI) portal group; in a first processor associated with afirst network interface and associated with the first iSCSI portalgroup, identifying data packets received over the first networkinterface as comprising a storage operation associated with at least oneof the plurality of storage drives managed by a second processor, andresponsively transferring the first data packets for handling by theprocessor having a second network interface associated with the firstiSCSI portal group; and in the second processor handling the storageoperation associated with the data packets as if the second networkinterface had received the data packets.
 16. The method of claim 10,further comprising: in at least one of the PCIe switches,interconnecting the processing modules of the data storage system withprocessing modules of another data storage system over at least oneexternal PCIe link, wherein the at least one external PCIe linkscomprises PCIe signaling carried over Serial Attached Small ComputerSerial Interconnect (SAS) connectors and cabling to interconnect thedata storage system to the other data storage system.
 17. The method ofclaim 10, further comprising: in at least one of the processors,presenting an associated one or more storage drives as at least one NVMExpress (NVMe) virtual logical unit number (VLUN) over a PCIe interfacethat communicatively couples the at least one of the processors and acomputing system external to the data storage system; in the at leastone of the processors receiving NVMe storage transactions transferred bythe computing system over the PCIe interface and handling the NVMestorage transactions for storage of data on the associated one or morestorage drives.
 18. A modular data storage system, comprising: aplurality of storage modules each comprising a plurality of solid statedrives and configured to communicate over a switched PeripheralComponent Interconnect Express (PCIe) fabric; a plurality of processingmodules each comprising one or more processors that are configured tomanage storage operations for a designated subset of the plurality ofstorage modules over the switched PCIe fabric; a plurality of networkinterfaces individually associated with ones of the processing modulesand configured to communicate with the associated ones of the processingmodules over the switched PCIe fabric; one or more PCIe switchesconfigured to establish the switched PCIe fabric over a plurality ofPCIe links to interconnect the storage modules to the processing modulesand interconnect the processing modules to the network interfaces; theprocessors configured to share a PCIe address space associated with theswitched PCIe fabric for transfer of received storage operations toappropriate ones of the processors that manage ones of the plurality ofstorage drives.
 19. The modular data storage system of claim 18,comprising: each of the network interfaces configured to be associatedwith a different Internet Protocol (IP) address and at least two of thenetwork interfaces configured to be associated with a first InternetSmall Computer System Interface (iSCSI) portal group; a first processorassociated with a first network interface and associated with the firstiSCSI portal group configured to identify data packets received over thefirst network interface as comprising a storage operation associatedwith at least one of the plurality of storage drives managed by a secondprocessor, and responsively transfer the first data packets for handlingby the processor having a second network interface associated with thefirst iSCSI portal group; and the second processor configured to handlethe storage operation associated with the data packets as if the secondnetwork interface had received the data packets.
 20. The modular datastorage system of claim 18, comprising: at least one of the processorsconfigured to present an associated one or more storage drives as atleast one NVM Express (NVMe) virtual logical unit number (VLUN) over aPCIe interface that communicatively couples the at least one of theprocessors and a computing system external to the data storage system;the at least one of the processors configured to receive NVMe storagetransactions transferred by the computing system over the PCIe interfaceand handle the NVMe storage transactions for storage of data on theassociated one or more storage drives.